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Abstract 


An  absence  of  guidance  on  designing  complex,  dynamic,  and  networked  systems  presents 
challenges  to  the  design  of  such  systems  to  maximize  overall  human-machine  system 
performance.  An  Intelligent  Adaptive  Interface  (IAI)  concept  and  associated  technologies  have 
been  developed  to  address  this  problem.  A  typical  IAI  is  driven  by  software  agents  that  can 
change  the  display  and  /or  control  characteristics  to  react  to  the  changes  of  mission  and  operator 
states  in  real  time.  The  work  reported  here  is  the  result  of  the  two  final  phases  of  a  three-year 
project  conducted  by  DRDC  Toronto.  This  project  investigated  the  efficacy  of  IAIs  in  a  multi- 
Uninhabited  Aerial  Vehicle  (UAV)  scenario.  The  IAI  was  modelled  as  part  of  the  UAV  tactical 
workstations  found  in  a  maritime  patrol  aircraft.  In  the  first  phase  of  the  project,  a  performance 
model  was  developed  to  compare  the  difference  in  mission  activities  with  and  without  IAI  agent 
aids.  The  simulation  results  revealed  that  the  control  of  multiple  UAVs  is  a  cognitively  complex 
task  with  high  workload.  With  the  augmentation  of  automation  agents,  operators  could  continue 
working  under  high  time  pressure,  resulting  in  critical  tasks  being  achieved  in  reduced  time.  To 
further  test  the  effectiveness  of  IAIs  and  validate  the  simulation  results,  a  prototype  IAI  multi¬ 
agent  experimental  environment  was  implemented  for  an  empirical  study.  Six  IAI  agent  function 
groups  have  been  integrated  into  the  UAV  operator  interfaces.  Operator’s  performance  was 
examined  with  and  without  IAIs  under  three  different  workload  conditions.  The  results  from  both 
objective  and  subjective  measures  verified  the  findings  of  the  simulation  research.  IAIs  facilitated 
a  significant  reduction  in  workload  and  an  improvement  in  situation  awareness.  This  research 
also  developed  preliminary  guidance  on  designing  IAI  systems. 
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Resume 


Un  probleme  qui  se  pose  en  ce  qui  conceme  la  commande  de  plusieurs  engins  telepilotes  est  la 
gestion  de  la  masse  d’ informations  necessaires  pour  appuyer  la  prise  de  decision  efficace.  De 
l’avis  des  operateurs  d’engins  telepilotes,  1’ amelioration  des  interfaces  operateur  entrainerait  des 
gains  importants  au  niveau  des  performances  et  de  l’efficacite  des  systemes.  Divers  niveaux 
d’automatisation  ont  ete  suggeres  pour  resoudre  le  probleme,  dont  l’utilisation  d’interfaces 
adaptatives  et  intelligentes  (IAI)  pour  l’aide  a  la  decision.  En  dotant  les  postes  de  commande  des 
engins  telepilotes  de  groupes  de  fonctions  d’automatisation,  les  1A1  ont  pour  but  de  gerer 
l’information  dynamiquement  et  de  foumir  la  bonne  information  aux  bonnes  personnes  au  bon 
moment,  pour  appuyer  la  prise  de  decision  efficace.  Les  travaux  decrits  dans  le  present  document 
sont  l’aboutissement  d’un  projet  de  trois  ans,  realise  par  R  &  D  pour  la  defense  Canada,  portant 
sur  l’efficacite  des  IAI  dans  un  scenario  de  commande  de  plusieurs  engins  telepilotes  dans  lequel 
les  IAI  sont  modelisees  comme  faisant  partie  des  postes  de  travail  tactiques  d’engins  telepilotes  a 
bord  d’un  avion  de  patrouille  maritime.  Un  modele  de  performance  a  ete  developpe  pour 
comparer  la  difference  entre  les  activites  de  mission  avec  et  sans  automatisation,  difference  qui  se 
reflete  dans  la  frequence  des  conflits  de  taches  et  le  temps  d’execution  des  taches.  Un  prototype 
d’environnement  experimental  d’lAI  a  ete  mis  en  oeuvre  pour  une  etude  empirique  a  intervention 
humaine.  Les  resultats  de  la  simulation  et  de  1’ experience  ont  montre  que  la  commande  de 
plusieurs  engins  telepilotes  est  une  tache  complexe  sur  le  plan  cognitif  avec  charge  de  travail 
elevee.  Avec  l’ajout  d’agents  d’automatisation,  les  IAI  ont  favorise  une  baisse  appreciable  de  la 
charge  de  travail  et  une  amelioration  de  la  connaissance  de  la  situation.  Les  operateurs  peuvent 
continuer  a  travailler  sous  de  fortes  contraintes  de  temps,  et  des  taches  critiques  peuvent  etre 
executees  en  moins  de  temps  qu’avec  des  interfaces  classiques. 
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Executive  summary 


Intelligent  adaptive  interfaces:  summary  report  on  design, 
development,  and  evaluation  of  intelligent  adaptive  interfaces 
for  the  control  of  multiple  UAVs  from  an  airborne  platform 

Ming  Hou;  Kobierski,  R.D.;  DRDC  Toronto  TR  2006-292;  Defence  R&D  Canada  - 

Toronto;  December  2006. 

Introduction:  The  deployment  and  control  of  Uninhabited  Aerial  Vehicles  (UAVs)  generate  an 
enormous  amount  of  data  that  will  become  even  more  complex  as  more  communication  channels 
are  engaged  between  air,  sea,  and  ground  for  joint  operations.  As  the  quantity  and  variety  of  those 
data  increase,  the  workload  of  UAV  operators  is  likely  to  increase  exponentially,  imposing  severe 
constraints  on  personnel  conducting  these  missions.  One  way  to  reduce  operator  demands  is  to 
convert  data  into  information  and  automatically  disseminate  it  to  the  right  decision-makers. 
Another  method  is  to  look  for  opportunities  to  limit  the  complexity  of  tasks  that  humans  perform 
when  controlling  UAVs.  A  third  approach  seeks  to  limit  the  number  of  tasks  to  be  performed. 

Feedback  from  UAV  operation  reports  indicates  that  there  is  a  need  for  improvement  in  the 
operator  interfaces  of  these  emerging  systems.  This  applies  to  effective  UAV  control  and  data 
management,  including  converting  data  into  information  and  efficiently  disseminating  the 
information  to  appropriate  users.  The  level  of  automation  (intelligent  and  adaptive  software) 
applied  to  the  decision-making  process  is  important  for  tactical  commanders  and  UAV  system 
managers.  Thus,  supporting  technologies  such  as  Intelligent  Adaptive  Interfaces  (IAIs)  requires 
investigation.  The  work  reported  here  is  the  result  of  a  three-year  project  that  investigated  the 
efficacy  of  IAIs  in  the  UAV  context.  The  selected  environment  involved  UAV  operations  in 
support  of  counter-terrorist  activities.  The  IAI  was  modeled  as  part  of  the  UAV  tactical 
workstations  for  a  modernized  Canadian  Maritime  Patrol  Aircraft  CP  140.  This  work  was  divided 
into  three  phases. 

Results:  Phase  I  produced  a  methodology  to  analyze  UAV  operations  in  a  mission  scenario.  The 
scenario  reflected  a  portion  of  a  Canadian  Forces  (CF)  UAV  experimental  program.  The 
analytical  results  were  used  to  develop  a  human-machine  task  network  model  that  was  then 
implemented  in  an  integrated  performance  network  modeling  environment.  The  model  has  two 
modes  for  operators  to  work  with  user  interfaces  controlling  multiple  UAVs.  One  mode  assumed 
that  operators  used  conventional  interfaces  to  control  multiple  UAVs.  The  other  mode  assumed 
that  operators  used  interfaces  with  IAI  automation  aiding.  The  difference  between  mission 
activities  with  and  without  IAI  aiding  was  reflected  in  the  time  to  complete  critical  task  sequences 
and  task  conflict  frequency.  The  simulation  revealed  that  the  use  of  a  control  interface  with  IAI 
mode  permitted  operators  to  complete  critical  task  sequences  in  reduced  time,  even  under  high 
time  pressure. 

Phase  II  focused  on  the  design  and  implementation  of  IAI  prototype  interfaces  which 
incoiporated  six  system  function  groups:  inter-crew  communications,  route  planning,  routing 
following,  screen  management,  data-link  monitoring,  and  UAV  sensor  selection.  A  synthetic 
environment  was  created  which  followed  the  North  Atlantic  Treaty  Organization  (NATO) 
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Standardization  Agreement  (STANAG)  4586  interface  software  protocol.  The  experimental 
environment  had  three  control  consoles  replicating  CP  140  tactical  compartment  workstations, 
with  a  set  of  appropriate  displays  and  controls  for  each  of  the  UAV  crew  members:  UAV  pilot, 
sensor  operator,  and  tactical  navigator.  The  experimental  environment  also  has  an  integrated 
video  and  audio  data  collection  suite  to  facilitate  empirical  assessment  of  IAI  concepts. 

In  Phase  III,  experiments  were  conducted  to  examine  operator  workload  and  interface  adaptability 
with  mock-up  UAV  control  consoles.  Eight  crews  (24  operational  CP  140  members)  participated 
in  the  experiment.  Each  crew  completed  a  two-day  experiment  that  assessed  operator  interfaces 
with  and  without  IAI  aiding.  The  results  showed  reduced  completion  time  for  critical  task 
sequences  in  the  IAI  mode.  There  was  also  a  significant  reduction  in  workload  and  an 
improvement  in  situation  awareness. 

Significance:  The  objective  of  this  project  was  to  demonstrate  IAI  capabilities  in  reducing 
workload  and  improving  operator  performance  during  the  deployment  of  UAVs.  Both  task 
network  simulation  and  human-in-the-loop  experimental  results  showed  that  the  use  of  IAIs  in 
advanced  operator  interfaces,  such  as  multiple  UAV  control  systems;  improved  the  effectiveness 
of  the  crew. 
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Introduction  ou  contexte  :  Le  deployment  et  la  commande  d’engins  telepilotes  generent  une 
quantite  enorme  de  donnees  qui  deviendront  meme  de  plus  en  plus  complexes  a  mesure  que  le 
nombre  de  canaux  de  communications  air-mer-sol  augmentera  pour  les  operations  interarmees. 
Avec  l’accroissement  de  la  quantite  et  de  la  variete  de  ces  donnees,  la  charge  de  travail  des 
operateurs  d’engins  telepilotes  risque  de  croitre  exponentiellement,  de  sorte  que  de  fortes 
contraintes  seront  imposees  au  personnel  executant  les  missions.  Une  (agon  de  reduire  les 
contraintes  imposees  aux  operateurs  consiste  a  convertir  ces  donnees  en  information  pertinente  et 
a  la  foumir  automatiquement  aux  decideurs  pertinents.  Une  autre  consiste  a  chercher  des 
occasions  de  limiter  la  complexite  des  taches  que  les  humains  accomplissent  pour  la  commande 
des  engins  telepilotes.  Une  troisieme  consiste  a  tenter  de  limiter  le  nombre  de  taches  a  accomplir. 

La  retroaction  a  la  suite  des  operations  menees  avec  les  engins  telepilotes  indique  qu’il  y  a  lieu 
d’ameliorer  les  interfaces  operateur  de  ces  nouveaux  systemes,  tant  en  ce  qui  conceme  la 
commande  reelle  des  engins  telepilotes  que  la  gestion  des  donnees,  y  compris  la  conversion  de 
ces  donnees  en  information  et  racheminement  efficace  de  cette  demiere  vers  les  utilisateurs 
appropries.  Le  niveau  d’automatisation  (logiciel  intelligent  et  adaptatif)  a  appliquer  aux  processus 
de  prise  de  decision  est  un  facteur  cle  tant  pour  les  commandants  tactiques  que  pour  les 
gestionnaires  de  systemes  d’engins  telepilotes.  Par  consequent,  il  faut  examiner  les  technologies 
d’appui  (p.  ex.  l’interface  operateur  adaptative  et  intelligente)  qui  font  appel  a  la  fois  aux 
interventions  des  operateurs  et  a  l’automatisation  pour  satisfaire  aux  exigences  des  missions.  Les 
travaux  decrits  dans  le  present  document  sont  l’aboutissement  d’un  projet  de  trois  ans,  realise  par 
R  &  D  pour  la  defense  Canada,  portant  sur  l’efficacite  d’interfaces  adaptatives  et  intelligentes 
(IAI)  dans  une  situation  operationnelle.  L’environnement  selectionne  a  donne  lieu  a  des 
operations  d’engins  telepilotes  visant  a  appuyer  des  activites  de  lutte  contre  le  terrorisme,  1’IAI 
etant  modelisee  pour  faire  partie  integrante  des  postes  de  travail  tactiques  d’engins  telepilotes  a 
bord  d’un  avion  de  patrouille  maritime  canadien  CP  140  modernise. 

Resultats  :  La  premiere  phase  du  projet  sur  les  IAI  a  permis  d’etablir  une  methode  d’analyse  des 
operations  d’engins  telepilotes  dans  un  scenario  de  mission  elabore  de  maniere  a  refleter  une 
partie  du  programme  de  renseignement,  de  surveillance  et  de  reconnaissance  du  littoral  atlantique 
des  Forces  canadiennes.  Les  resultats  de  l’analyse  ont  ete  utilises  pour  elaborer  un  modele  de 
performance  qui  a  ensuite  ete  mis  en  oeuvre  dans  un  environnement  de  modelisation  de  reseau  de 
performance  integre.  Le  modele  a  ete  utilise  dans  deux  modes  :  dans  le  premier,  on  considerait 
que  les  operateurs  utilisaient  une  interface  classique  pour  commander  plusieurs  engins  telepilotes; 
dans  le  second,  on  considerait  que  l’automatisation  etait  realisee  a  l’aide  d’une  IAI.  La  difference 
entre  les  activites  de  mission  avec  et  sans  automatisation  s’est  manifestee  dans  le  temps 
necessaire  a  1’ execution  de  sequences  de  taches  critiques  et  dans  d’autres  mesures  de  la 
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performance.  La  simulation  a  montre  que  l’utilisation  d’une  console  de  commande  a  laquelle  un 
mode  IAI  est  integre  a  permis  aux  operateurs  de  continuer  a  travailler  sous  de  fortes  contraintes 
de  temps  et  d’atteindre  les  objectifs  de  niveau  superieur  dans  un  delai  reduit. 

La  deuxieme  phase  etait  axee  sur  la  conception  et  la  mise  en  oeuvre  de  prototypes  d’lAI  integrant 
cinq  groupes  de  fonctions  d’lAI  :  communications  entre  equipages,  planification  de  trajectoire, 
gestion  d’ecran,  surveillance  de  liaison  de  donnees  et  selection  de  capteurs  d’engins  telepilotes. 
On  a  cree  un  environnement  synthetique  conforme  au  protocole  de  logiciel  d’ interface  STANAG 
4586  de  l’OTAN.  L’ environnement  experimental  comprenait  trois  consoles  de  commande 
reproduisant  les  postes  de  travail  de  compartiment  tactique  du  CP  140,  avec  un  ensemble 
d’affichages  et  de  commandes  appropries  pour  le  pilote  d’engin  telepilote,  l’operateur  de  capteurs 
et  le  navigateur  tactique.  Cet  environnement  permet  aussi  la  video  integree  et  la  collecte  de 
donnees  audio,  de  sorte  qu’il  est  possible  de  faire  une  evaluation  empirique  des  concepts  d’lAI 
elabores  dans  la  premiere  phase. 

La  troisieme  phase  consistait  en  l’execution  d’experiences  visant  a  examiner  la  charge  de  travail 
des  operateurs,  la  complexite  des  taches  et  l’adaptabilite  d’interface  lorsque  des  consoles  de 
commande  de  maquettes  d’engins  telepilotes  sont  utilisees.  Huit  equipages  (au  total  24  membres 
operationnels  du  CP  140)  ont  participe  a  1’  experience.  Chaque  equipage  a  realise  une  experience 
de  deux  jours  visant  a  evaluer  subjectivement  et  objectivement  les  systemes  de  commande  avec  et 
sans  aide  IAI.  Les  resultats  ont  montre,  dans  le  mode  IAI,  une  amelioration  importante  de  la 
performance  de  l’equipage  quant  au  temps  requis  pour  l’execution  de  sequences  de  taches 
critiques;  en  outre,  on  a  observe  une  reduction  appreciable  de  la  charge  de  travail  et  une 
amelioration  de  la  connaissance  de  la  situation. 

Importance  :  L’objectif  du  projet  consistait  a  determiner  si  les  fonctions  d’lAI  permettraient  de 
reduire  la  charge  de  travail  et  d’ameliorer  la  performance  des  operateurs  durant  le  deployment 
d’engins  telepilotes.  Les  resultats  de  la  simulation  de  reseau  de  taches  et  de  l’experience  a 
intervention  humaine  ont  montre  que  la  commande  de  plusieurs  engins  telepilotes  est  une  tache 
complexe  sur  le  plan  cognitif  avec  charge  de  travail  elevee.  On  a  conclu  que  l’utilisation  d’lAI 
dans  les  systemes  d’operateur  avances,  par  exemple  les  systemes  de  commande  de  plusieurs 
engins  telepilotes,  presenterait  de  reelles  possibilites  d’ amelioration  de  l’efficacite  de  l’equipage. 
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Introduction  and  motivation 


A  goal  for  the  design  of  complex,  dynamic,  and  networked  systems  (e.g.,  military  systems  used  in 
network  centric  warfare)  is  to  maximize  overall  human-machine  system  performance.  A  lack  of 
guidance  for  designing  these  systems  and  their  characteristics  (e.g.,  complexity,  dynamics,  or 
information  overload)  generates  challenges  to  achieving  this  goal.  However,  advances  in  human 
factors  engineering,  artificial  intelligence,  and  related  disciplines  provide  numerous  potentials 
that  can  be  leveraged  for  effective  design.  One  advanced  interface  design  concept  that  attempts 
to  do  so  is  the  Intelligent  Adaptive  Interface  (IAI).  An  IAI  is  an  operator  interface  that  changes 
the  display  and/or  control  characteristics  of  human-machine  systems  to  react  to  external  events 
(mission  and  operator  states)  in  real  time.  A  typical  IAI  is  driven  by  software  agents  (automation) 
that  provide  aids  to  satisfy  the  decision-making  and  action  requirements  of  operators  under 
different  levels  of  workload.  It  is  designed  to  present  the  right  information  or  action  sequence 
proposals,  or  performs  actions,  in  the  right  format,  and  at  the  right  time.  Although  there  are  many 
different  definitions  of  IAI  in  the  literature  and  many  different  names  (Intelligent  User  Interface, 
Maybury,  1998;  Intelligent  Interface  Technology,  Benyon,  2000;  and  User  Adaptive  Systems, 
Jameson,  2003),  the  key  element  is  that  an  IAI  reacts  adaptively  to  the  external  events  including 
both  mission  and  operator  states. 

An  early  concept  related  to  the  IAI  is  the  crew  adaptive  cockpit  (Reising,  1979).  The  goal  here 
was  to  migrate  tasks  from  operators  and  help  them  make  tactical  and  strategic  decisions.  Operator 
interface  technologies  exist  in  a  number  of  guises,  from  conventional  automation  to  intelligent 
adaptive  aiding.  Conventional  automation  was  designed  to  replace  human  control  and  decision¬ 
making.  However,  due  to  its  nature  of  being  out  of  the  loop  of  the  external  events  in  some 
situations,  it  may  be  difficult  for  operators  to  maintain  situation  awareness  (Billings,  1991;  Cook, 
Woods,  McColligan,  &  Howie,  1990;  Endsley,  1996;  Endsley  &  Kiris,  1995;  Moray,  1986; 
Norman,  1990)  and  vigilance  (Moray,  1986).  It  may  make  it  difficult  for  operators  to  calibrate 
the  automation’  capabilities  to  the  current  state  (Lee  &  Moray,  1992;  Parasuraman  &  Mouloua, 
1996;  Parasuraman,  Mouloua,  Molly,  &  Hilbum,  1993).  In  addition,  it  may  cause  loss  of  skills, 
over-trust  (i.e.,  complacency),  lack  of  trust  (i.e.,  scepticism),  and  increased  system  complexity 
(Parasuraman,  Sheridan,  &  Wickens,  2000;  Sarter  &  Woods,  1994;  Will,  1991),  and  other 
problems.  On  the  other  hand,  adaptive  automation  offers  an  alternative  design  approach  to 
allocate  systems  functions  to  automation  and  a  human  operator  dynamically  over  time.  The 
relationship  between  automation  and  operator  should  be  flexible  and  context  dependent  to 
optimize  overall  system  performance  (e.g.,  Kaber  &  Riley,  1999;  Parasuraman,  1987;  Rouse, 
1977;  Scerbo,  1996). 

In  contrast  to  conventional  automation,  the  provision  of  adaptive  automation  aiding  is  not  pre¬ 
determined  at  the  design  stage  and  the  task  allocation  to  human  or  system  is  not  fixed.  Thus,  an 
adaptive  automation  system  is  dynamic  in  nature.  The  loci  of  control  change  by  taking 
advantages  of  the  differences  between  the  abilities  of  humans  and  machines.  In  providing  this 
dynamic  or  adaptive  support,  the  perceived  loss  of  control  with  static  automation  can  be  reduced 
as  human  operator  remains  “in-the-loop”  (e.g.,  Hilbum,  Molloy,  Wong,  &  Parasuraman,  1993; 
Hilbum,  Joma,  Byme,  &  Parasuraman,  1997;  Kaber,  Riley,  Tan,  &  Endsley,  2001;  Kaber, 
Wright,  Prinzel,  Clamann,  2005;  Parasuraman,  Mouloua,  Molly,  &  Hilbum,  1993;  Scerbo,  1996). 
However,  adaptive  automation  does  introduce  more  complexity  in  the  allocation  of  tasks  that  can 
result  in  new  problems  such  as  being  out  of  system  functional  state  and  automation  failure 
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detection.  A  key  design  issue  is  to  optimise  the  triggering  conditions  for  task  re-allocation 
(e.g.,  by  monitoring  behaviour,  physiological  changes,  and/or  situation  events). 

At  the  highest  level  of  maturity,  adaptive  aiding  systems  “intelligently”  augment  and  enhance 
human  judgment  and  responsibility.  These  systems  can  adapt  to  dynamic  requirements  of 
external  events:  both  operator  and  mission  states.  These  adaptive  aiding  systems  can  be 
considered  to  be  “intelligent”  as  they  exhibit  behaviours  that  are  consistent  with  human 
intelligence  (Taylor  &  Reising,  1998):  for  example,  being  goal  driven;  capable  of  actively 
collecting  information;  capable  of  reasoning  at  multiple  levels;  capable  of  learning  from 
experience;  and  capable  of  context-sensitive  communication  with  the  operator  (i.e.,  they  are 
lAIs).  The  “intelligence”  of  such  1A1  systems  is  derived  from  a  functional  architecture  that 
couples  real-time  mission  analysis  with  real-time  monitoring  of  the  psychological,  physiological, 
and  behavioural  state  of  the  operator.  The  information  from  these  modules  can  then  be  used  to 
mediate  the  timing,  salience,  and  autonomy  operator  aids.  The  functional  architecture  should 
have  the  following  attributes:  a  model  of  human  decision-making  and  control  abilities;  the  ability 
to  monitor  operator  performance  and  workload  through  behavioural  and  physiological  indices; 
and  the  ability  to  predict  operator  expectations  and  intentions  with  reference  to  embedded 
knowledge  of  mission  plans  and  goals. 

There  is  some  empirical  evidence  supporting  the  benefits  of  IAIs.  A  PACT  (Pilot  Authorization 
and  Control  of  Tasks)  system  uses  a  system  functional  architecture  to  design  IAIs  to  reduce 
operator  workload  and  increase  situation  awareness.  It  is  part  of  the  Cognitive  Cockpit  system 
(Banbury,  Bonner,  Dickson,  Howells,  &  Taylor,  1999;  Taylor,  Bonner,  Dickson,  Howells,  Miller, 
Milton,  Pleydell-Pearce,  Shadbolt,  Tennison,  &  Whitecross,  2002)  which  couples  a  situation 
assessment  module  with  a  pilot  state  estimator  to  drive  intelligent  adaptable  Pilot-Vehicle 
Interfaces  (PVI).  The  resultant  PVI  adaptations  take  the  form  of  readily  inteipreted  information, 
warning  messages,  advisory  displays  and  control  feedback  that  allows  the  pilot  to  remain  in  full 
command  of  the  aircraft.  The  PACT  system  also  includes  the  notion  of  the  Rotorcraft  Pilot’s 
Associate  project  (Miller  &  Hannen,  1998)  which  took  the  strategy  to  migrate  tasks  from 
operators  to  automation  agents  to  improve  situation  awareness.  Another  IAI  use  is  a  metaphor  of 
controlling  automation  function  in  the  ‘playbook’  (Miller,  2000,  Miller,  2003;  Miller,  Goldman, 
Funk,  Wu,  &  Pate,  2004).  In  this  IAI  system,  there  are  a  number  of  ‘plays’  available  to  the 
operator  in  which  the  role  of  the  automation  and  operator  for  a  particular  task  has  been  pre- 
configured  (e.g.,  the  system  provides  advice  if  authorized  by  the  operator).  Although  these 
studies  have  provided  a  valuable  starting  point  for  prototyping  and  testing  any  other  IAI  systems 
for  the  purpose  of  workload  reduction  and  situation  awareness  improvement,  the  Cognitive 
Cockpit  did  not  provide  fully  implemented  models  and  the  “playbook”  challenges  the  operator’s 
memory  and  effectively  using  many  “plays”  in  a  time-critical  situation.  For  experimental  purpose, 
there  is  still  a  need  to  identify  an  experimental  technique.  The  technique  will  allow  for  testing  of 
observable  IAI  functions  without  a  full-scale  implementation.  For  design  purpose,  there  is  not 
any  guidance  on  IAI  systems. 

In  order  to  address  the  issues  above,  Defence  Research  &  Development  Canada  (DRDC)  - 
Toronto  started  a  multi-year  project  for  the  development  and  evaluation  of  IAIs  for  multiple 
Uninhabited  Aerial  Vehicle  (UAV)  control.  The  aim  of  the  project  was  not  only  to  design, 
develop,  demonstrate,  and  prioritize  enabling  IAI  technologies,  but  also  to  develop  design 
guidelines  for  IAI  systems.  These  technologies  can  be  applied  to  advanced  operator  interfaces 
which  will  support  reduced  manning  and  enhanced  performance  in  complex  military  systems, 
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particularly  multiple  UAV  control  from  an  airborne  platform.  This  project  laid  the  foundation  for 
the  production  of  preliminary  design  guidelines  for  IAIs  in  this  context. 

1AI  technologies  should:  a)  improve  human-machine  system  performance,  b)  reduce  operator 
workload,  and  thus  c)  enhance  situation  awareness.  In  order  to  test  these  hypotheses,  the  IA1 
project  was  conducted  in  three  phases:  IAI  concept  development,  interface  prototyping,  and 
experimentation.  Phase  I  of  the  project  involved  an  analysis  of  UAV  operations  in  a  mission 
scenario  to  support  counter-terrorism  activities.  The  scenario  involved  operations  with  the  IAI 
modeled  as  part  of  the  UAV  tactical  workstations  of  a  modernized  Canadian  CP  140  Maritime 
Patrol  Aircraft.  In  the  scenario,  the  CP  140  crew  took  over  the  UAV  operation  in  the  role  of  UAV 
Pilot  (UP),  UAV  Sensor  Operator  (UO),  and  Tactical  Navigator  (TN)  in  the  tactical  compartment 
of  the  aircraft.  The  analytical  results  were  used  to  develop  an  operator-machine  performance 
model  that  was  implemented  in  an  integrated  performance  modeling  environment.  The  model  has 
two  modes.  One  mode  assumed  that  operators  used  conventional  interfaces  to  control  multiple 
UAVs.  The  other  assumed  that  operators  used  interfaces  with  IAI  automation  aiding.  The 
difference  between  mission  activities  with  and  without  IAI  aiding  was  reflected  in  the  time  to 
complete  critical  task  sequences  and  task  conflict  frequency.  The  simulation  revealed  that  the  use 
of  a  control  interface  with  IAI  mode  permitted  operators  to  complete  critical  task  sequences  in 
reduced  time,  even  under  high  time  pressure.  Hou  &  Kobierski  (2006)  and  CMC  Electronics  Inc. 
(2004)  described  the  simulation  work  done  in  the  first  phase. 

This  report  summarizes  Phases  II  and  III  of  the  IAI  project.  These  two  phases  focused  on  the 
design  and  implementation  of  IAI  prototype  interfaces,  and  on  experimentation  that 
investigated  IAI  efficacy.  Further,  preliminary  design  guidelines  were  developed  to  guide  the  IAI 
system  design. 
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2  IAI  Prototype  and  mission  scenario 


An  experimental  Synthetic  Environment  (SE)  was  designed  and  developed  using  the  North 
Atlantic  Treaty  Organization  (NATO)  Standardization  Agreement  (STANAG)  4586  (CDL,  2005) 
interface  software  protocol.  The  SE  had  three  control  consoles  replicating  CP  140  tactical 
compartment  multifunction  workstations  consistent  with  the  UAV  crew  positions  used  in  Phase  I 
(concept  development).  The  workstations  were  designed  to  communicate  with  virtual  UAVs 
through  software  interfaces.  Each  had  a  set  of  displays  and  controls  appropriate  for  the  UP,  UO, 
and  TN  positions.  The  experimental  environment  could  also  collect  video  and  audio  data.  The 
experimental  protocol  was  approved  by  DRDC  Eluman  Research  Ethics  Committee  in 
2005  (see  Annexes  A-F). 

2.1  Prototype  interface  layout 

The  CP  140  Subject  Matter  Experts  (SMEs)  noted  that  in  an  operational  subsystem  UAV 
operators  occupy  the  three  rear  positions  of  the  CP  140  tactical  compartment  (illustrated  in 
Figure  1).  Thus  the  experimental  environment  has  been  designed  to  match  these  three  positions. 
The  overall  configuration  included  UP,  UO,  and  TN  workstations  as  shown  in  Figure  2.  Each 
workstation  consisted  of  a  display  screen,  keyboard.  Programmable  Entry  Panel  (PEP), 
trackball/mousc,  and  joystick  (for  UP  and  UO  only).  The  UP  and  the  UO  were  seated  next  to 
each  other  and  shared  a  console,  as  illustrated  in  Figure  3.  A  shared  display  screen  (i.e.,  the 
middle  display  in  Figure  3)  was  placed  between  the  operators.  This  display  was  a  modified 
version  of  the  TN’s  Tactical  Plot  (TACPLOT)  that  showed  various  contacts  and  their  associated 
track  numbers.  With  this  set-up,  the  shared  display  could  facilitate  communications  between  UP, 
UO,  and  TN  within  the  same  contexts. 

A  TN’s  primary  display  is  shown  in  Figure  4,  which  was  designed  to  communicate  tactical 
information.  The  TN  display  illustrated  the  supervision  and  management  of  tactical  situations  as 
well  as  the  consistencies  with  the  TN’s  main  role  as  coordinator  of  the  CP  140  UAV  crew. 

The  UP’s  primary  display  is  shown  in  Figure  5.  This  provided  the  information  necessary  to  pilot 
UAVs.  The  basic  layout  allowed  operators  to  display  a  mini-TACPLOT  and  pilot  camera  view 
for  up  to  two  UAVs.  All  pilot  camera  views  had  Eleads-Up-Display  (E1UD)  style  symbology 
(superimposed)  to  show  critical  flight  data.  Since  some  UAVs  do  not  have  a  pilot 
camera,  a  solid  background  replaced  the  pilot  camera  image,  although  the  same  ElUD-style 
symbology  was  shown. 

The  UO’s  primary  display  is  shown  in  Figure  6.  This  was  designed  to  allow  the  UO  to  manage 
and  extract  information  from  many  sensors.  Operators  could  flexibly  manipulate  the  layout  of  the 
main  display  area.  Information  about  the  instantaneous  direction  of  the  turret  with  respect  to  the 
current  UAV  heading,  sensor  elevation  angle,  and  zoom  setting  was  superimposed  on  the 
sensor  video. 
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Figure  1:  CP  140  Tactical  compartment  layout  with  Tactical  Navigator  (TN),  UAV  Pilot  (UP), 
UAV  Sensor  Operator  (UO),  and  Navigator  Communicator  (NAVCOM)  positions 


Figure  2:  Experimental  environment  showing  three  UAV  control  workstations  and  NAVCOM 

workstation  (to  the  right) 
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Figure  3:  UP  and  UO  positioned  at  the  Aft  rearward-facing  workstations 


Figure  4  .'Primary  display  for  TN 
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Figure  5:  Primary’  display  for  UP 


Figure  6:  Primary  display  for  UO 


2.2  IAI  software 

The  IAI  agents  were  system  functional  components  of  the  UAV  control  SE.  They  provided 
decision  support  to  the  experimental  participants  and  took  over  some  crew  tasks  with 
high  workload. 
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The  IAI  agents  followed  this  sequence: 

•  Step  1 :  Gather  status  information  about  all  active  UAVs,  their  tracks,  and  the  current  display 
configuration. 

•  Step  2:  Analyse  the  information  with  respect  to  the  SMEs’  pre-defmed  rules  and  determine 
which  events  have  occurred. 

•  Step  3:  Prioritize  the  events  according  to  pre-defmed  rules. 

•  Step  4:  Execute  pre-defmed  tasks  for  each  identified  event  following  the  prioritization  order. 

Depending  on  the  results,  1A1  agents  could  actively  support  the  operators  in  the  following  tasks 
identified  by  the  SMEs  in  this  context: 

•  Task  1:  Route  Planning.  If  a  UAV  was  used  to  investigate  an  unknown  or  hostile  contact, 
the  agent  would  compute  the  most  direct  route  and  activate  that  route  for  the  UAV.  The 
allocation  of  tracks  to  UAVs  was  based  on  a  search  for  closest  unknown  or  hostile  contact. 
Additional  logic  ensured  that  no  more  than  one  UAV  could  be  engaged  on  a  single  unknown 
contact.  More  than  one  UAV  could  be  engaged  on  hostile  contacts. 

•  Task  2:  Route  Following.  The  agent  would  pilot  the  UAV  on  the  active  route.  This 
included  flight  altitude,  speed  management,  and  self-preservation  in  close  proximity  to  the 
track.  The  agent  entered  an  orbital  flight  pattern  around  the  track  once  the  UAV  reached 
sensor  identification  range. 

•  Task  3:  Screen  Management.  The  shared  TACPLOT  was  agent-managed  (according  to  pre¬ 
defined  rules)  whenever  new  high-priority  events  occurred.  This  included  automatically 
panning  the  TACPLOT  to  a  location  of  interest  and  zooming  in  or  out. 

•  Task  4:  Inter-crew  Communications.  All  observations  on  UAVs  track  relationships  were 
reported  by  the  agent  to  the  crew  via  the  IAI  message  window.  Thus,  the  crew  did  not  need 
to  make  or  confirm  theses  observations. 

•  Task  5:  Sensor  Management.  Once  a  UAV  was  close  enough  to  a  track  to  engage  an 
Electrical  Optic  (EO)  sensor,  the  agent  would  take  over  sensor  management.  This  included 
pointing  the  sensor  and  establishing  a  stable  lock  on  the  moving  target  once  the  track  was 
within  visual  range. 

•  Task  6:  Data  Link  Monitoring.  The  agent  monitored  the  flight  pattern  (and  other  status  of 
the  UAV)  to  determine  whether  the  data  link  was  working.  If  not,  the  agent  would 
immediately  inform  the  crew. 

Six  software  agents  were  designed  and  implemented  in  the  IAI  prototype  interfaces  as 
multi-agent  subsystems. 

To  allow  agents  to  communicate  events  and  actions  to  the  crew  quickly  and  reliably,  the  IAI 
Graphical  User  Interface  (GUI)  showed  IAI  messages  on  the  primary  display.  Figure  7  shows  an 
example  of  an  IAI  message  window  used  on  the  operators’  primary  display.  The  IAI  message 
window  is  also  shown  in  the  bottom  left  comer  of  Figures  4,  5  and  6.  The  IAI  message  window 
shows  all  active  UAVs,  allocated  tracks,  and  information  about  how  the  IAI  agent  supported  the 
UAV.  On  the  TACPLOTs,  text  was  added  to  the  UAV  icon  as  illustrated  in  Figure  8.  This 
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identified  the  contact  designated  by  an  IAI  agent  and  whether  the  agent  was  exercising  UAV 
flight  control  (“P”)  and/or  sensor  geotracking  (“S”). 


MESSAGES 


UAV  TRACK  STATUS 

9  164  NAV/EO 


Figure  7:  IAI  message  window 


Figure  8:  IAI  readouts  in  TACPLOT 


2.3  Mission  scenario 

To  evaluate  IAI  prototype  interfaces,  a  counter-terrorism  mission  scenario  was  developed  and  set 
in  the  year  2011.  The  Commonwealth  of  Nations  had  chosen  St  John’s,  Newfoundland  as  the  site 
for  the  bi-annual  Commonwealth  Heads  of  Government  Meeting  (CHOGM).  The  Canadian 
Forces  (CF)  was  to  provide  security  for  the  meeting.  At  1745  hrs,  August  11,  2011,  British 
Intelligence  relayed  information  about  a  Lethal  Medium  Range  UAV,  a  potential  threat  to  the 
CHOGM.  This  device  could  be  launched  from  a  van-sized  steel  container.  The  intelligence 
groups  suspected  that  the  Lethal  UAV  had  been  launched  from  a  boat  as  far  as  240  nautical  miles 
(nm)  away.  The  group  suspected  of  fielding  the  weapon  had  obtained  a  quantity  of  plutonium 
from  a  nuclear  power  plant.  A  UAV  carrying  a  plutonium  “dirty  bomb”  would  cause  many 
casualties  and  render  the  targeted  region  potentially  uninhabitable  for  years. 
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In  the  meantime,  there  was  a  fisheries  patrol  southeast  of  St.  John’s,  and  about  200  vessels  were 
in  the  vicinity  of  the  Grand  Banks  on  the  east  coast  of  Canada.  The  Canadian  frigate  HMCS 
Halifax  was  in  the  region  with  two  Vertical  Take-Off  UAVs  (VTUAV)  and  a  Maritime 
Helicopter  (MH).  A  CP140  patrol  aircraft  equipped  with  16  Mini  UAVs  and  a  sensor  suite  was 
overhead.  Figure  9  illustrates  the  scenario. 

The  scenario  began  at  1800  hrs,  after  the  CP  140  crew  had  received  information  that  there  was  a 
possible  terrorist  threat  to  the  CHOGM.  The  crew  was  tasked  to  search  for  a  vessel  carrying  a 
launch  container  (approximately  10  ft  x  8  ft  x  20  ft).  Intelligence  reports  had  suggested  that  the 
threat  might  come  from  a  trawler-sized  vessel.  The  UAV  crew  was  provided  with  a  VTUAV 
from  HMCS  Halifax.  Once  the  VTUAV  cleared  its  mother  ship,  HMCS  Halifax  made  ready  and 
launched  the  MH.  The  mission  was  to  investigate  a  concentration  of  vessels  to  the  south.  The 
ship’s  crew  knew  that  recovery  of  VTUAV  1  would  be  necessary  at  approximately  the  same  time 
that  the  MH  returned.  However,  this  was  declared  after  the  VTUAVs  and  the  MH  were  airborne. 

At  1843  hrs,  contact  was  lost  with  VTUAV  1  as  it  approached  a  vessel  under  investigation.  The 
CP  140  launched  three  Mini  UAVs  over  the  contact  area  and  warned  other  airborne  units  to  avoid 
the  possible  threat.  The  Mini  UAVs  approached  and  started  to  investigate  the  vessel.  At  the  same 
time,  HMCS  Halifax  made  the  best  possible  speed  to  the  same  location.  VTUAV  2  was  also 
directed  towards  the  suspicious  boat,  and  its  control  was  passed  to  the  CP  140  crew. 

At  approximately  1850  hrs,  a  Mini  UAV  transmitted  an  image  of  men  working  on  the  trawler’s 
fo’c’s’le.  A  large  storage  container  was  exposed.  The  CP140  used  the  EO  sensor  of  VTUAV  2 
and  Mini  UAVs  to  observe.  The  container  was  opened  to  expose  a  Jet  Assist  Take-Off  UAV. 
Minutes  later,  the  lethal  UAV  was  launched.  Two  CF18s  were  ordered  to  attack  the  now 
identified  terrorist  boat.  Assisted  by  a  laser  UAV  controlled  from  the  CP  140  the  terrorist  boat 
was  destroyed  prior  to  launching  a  second  lethal  UAV.  At  1900  hrs,  the  experimental  scenario 
ended.  The  CP  140  crew  initiated  a  search  for  the  lethal  UAV,  which  was  tracking  toward 
St  John’s. 
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MALE  UAV  Patrol  Area 


SUCH 


MALE  UAV 


Canadian  CP  140  Aircraft 


Group  of  Fishing  Vessels 
Approximately  75  boats 


CP140  Patrol  Area 


CPF  Patrol  Area 


VTUAV  1 


VTUAV 2 


HMCS  Halifax 


Group  of  Fishing  Vessels 
Approximately  75  boats / 


50 

Statute  Miles 


Figure  9:  Grand  Banks  overview  at  1800  hrs  (MALE  implies  Medium  Altitude  Long 

Endurance) 
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3  Experimental  design 


To  determine  if  IAIs  reduce  operator  workload  and  improve  Situation  Awareness  (SA)  and 
performance,  a  human-in-the-loop  empirical  investigation  on  the  efficacy  of  IAIs  was  conducted. 
The  IAI  multi-agent  system  supported  six  IAI  tasks  as  described  above  in  Section  2.2.  The 
evaluation  was  conducted  with  two  different  operator  interface  modes  (IAI  features  on  and  off). 
Besides  the  interface  condition,  the  other  experimental  variable  was  workload  which  had  low, 
medium,  and  high  levels  built  in  as  three  parts  of  the  scenario,  respectively. 

3.1  Participants 

The  roles  of  UAV  operators  as  experiment  participants  were  assigned  positions  as  described  in 
Section  2.1: 

Tactical  Navigator  (TN).  The  TN  was  the  mission  commander  and  responsible  for  managing  the 
use  of  all  resources  to  accomplish  the  mission  goals. 

UAV  Pilot  (UP).  The  UP  was  responsible  the  deployment,  management,  and  control  of 
individual  UAVs. 

UAV  Sensor  Operator  (UO).  The  UO  was  responsible  for  the  selection  and  management  of  the 
EO  sensors  and  the  interpretation  of  sensor  data. 

Eight  UAV  crews  were  recruited  as  volunteers  from  the  CP  140  community  at  the  Canadian 
Forces  Bases  (CFB)  Comox  and  Greenwood.  Each  crew  had  three  members  taking  UAV  control 
positions:  UP,  UO,  and  TN.  The  assignment  of  roles  was  based  on  the  amount  of  experience  in 
the  aforementioned  positions,  with  crewmembers  being  assigned  to  the  position  with  which  they 
had  the  most  experience.  All  participants  were  male,  and  the  age  ranged  from  26  to  52  years 
(Mean  =  40.1  years).  All  had  operational  experience  with  the  CP140,  and  the  experience  ranged 
from  half  a  year  (120  flying  hours)  to  20  years  (6300  flying  hours)  (Mean  =  8.2  years  or  2679 
hours).  All  participants  were  “fit  to  fly”  for  the  experiments  in  the  simulated  CP  140  tactical 
compartment.  The  UAV  crews  were  supplemented  by  an  experimental  staff  member  who  played 
the  role  of  Navigator  Communicator  (NAVCOM).  The  NAVCOM’s  primary  role  was  to  liaise 
with  the  experimental  staff  and  the  crew  (experiment  participants).  As  the  experimental  scenario 
unfolded,  the  NAVCOM  communicated  to  the  subject  crew  (i.e.,  UP,  UO,  and  TN)  the  “taskings” 
from  the  Maritime  Operations  Centre  (MOC)  at  Maritime  Forces  Atlantic,  in  Flalifax. 

3.2  Apparatus 

The  experimental  environment  consisted  of  multiple  hardware  and  software  components.  The 
major  hardware  components  were: 

1 .  Participants'  consoles  and  physical  layout  of  the  experimentation  area; 

2.  Participants’  prototype  GUI  display;  and 
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3.  Integrated  video  and  audio  data  collection  suite. 

The  experimental  environment  is  depicted  in  Figure  2.  This  figure  also  shows  the  NAVCOM’s 
position  relative  to  the  UP,  UO,  and  TN.  The  interfaces  for  these  three  crewmembers  are 
described  in  Section  2.1  and  illustrated  in  Figures  3,  4,  5,  6.  The  TN  and  the  NAVCOM  shared  a 
console,  but  all  the  scenario  details  were  shown  on  the  NAVCOM’s  display  and  therefore  were 
blocked  from  the  TN’s  line  of  sight  by  a  physical  barrier. 

All  video  data  were  captured  using  either  Pelco  TM  high-resolution  security  cameras  or  using 
Extron  TM  scan  converters,  which  allowed  video  recording  of  all  participants’  monitors.  All 
video  data  were  relayed  through  a  compatible  personal  computer  running  Pelco  TM  security 
software.  They  were  displayed  on  a  21”  Silicon  Graphics  Cathode  Ray  Tube  (CRT)  monitor.  All 
audio  data  captured  were  relayed  through  an  aircraft  quality  communication  system  to  the  Digital 
Video  Recorder.  They  were  linked  to  each  of  ten  video  input  channels.  There  were  two  audio 
channels  on  the  communication  system.  The  main  channel  was  the  crew  channel,  on  which  all 
parties  could  hear  all  participants’  communications.  The  secondary  channel  was  the  experimental 
channel,  on  which  only  the  experimental  staff  (including  NAVCOM)  could  hear  one  another. 

The  experimental  environment  had  also  multiple  software  components  communicating  within  a 
Microsoft  Windows  XP  environment.  The  major  software  components  were: 

1.  An  IAI  multi-agent  system  embedded  in  the  operators’  interfaces; 

2.  A  software  package  that  completed  data  collection  of  operator  keystrokes  and  vehicle 
motions; 

3.  A  three-dimensional  world  through  which  the  UAVs  flew  and  were  viewed  by  the 
participants  through  the  UAV-mounted  video  cameras; 

4.  A  software  package  that  simulated  rotary  wing  and  fixed  wing  air  vehicle  flight  dynamics  and 
autonomous  surface  vehicle  (boat)  motion;  and 

5.  A  NATO  STANAG  4586  compatible  software  that  allowed  communication  between  the 
consoles  and  the  simulated  UAVs. 

The  IAI  multi-agent  system  was  a  process  software  component  embedded  in  the  Data 
Management  System  (DMS)  of  the  UAV  IAI  SE.  When  all  IAI  agents  (interface  function 
groups)  were  switched  off  (IAI  OFF),  the  interface  was  simply  the  conventional  interface  without 
the  aid  of  the  IAI  multi-agent  system.  When  all  IAI  agents  were  switched  on  (IAI  ON),  the 
various  function  groups  coded  into  the  interface  software  were  activated.  The  DMS  was  the 
central  data  processing  component  of  the  UAV  IAI  SE  and  served  as  a  protocol  gateway  between 
the  workstations  and  the  external  simulation  components.  Figure  10  illustrates  the  DMS 
architecture. 

The  DMS  maintained  a  synchronized  situational  data  repository  of  all  relevant  information 
related  to  the  UAVs,  the  CP140  ownership,  and  the  surface  and  air-bome  tracks.  The  DMS 
processed  and  organized  pertinent  information  for  efficient  consumption  by  the  workstations  so 
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that  the  workstation  operators  could  relay  UAV  control  commands  to  the  simulated  UAVs  and 
change  workstation  display  configurations. 

All  entity  motion  data  were  recorded  on  a  Windows  XP  computer  hard  drive.  This  information 
included  time-sequenced  player  positions  to  allow  post  experiment  plotting  of  routes  for  all 
entities  and  a  record  of  all  information  relayed  between  the  workstation  and  the  vehicle  motion 
simulation  software  STRIVE® (CAE,  2005). 

STRIVE”  was  modified  from  controlling  one  UAV  to  concurrently  controlling  multiple  UAVs. 
The  control  of  a  number  of  UAVs  and  their  sensors  followed  NATO  STANAG  4586  standard 
through  the  simulated  mission  computer  (with  dual  or  single  processor  and  32  MB  OpenGL 
graphic  cards).  The  sensor  views  of  the  SE  relayed  through  STANAG  4586  were  provided 
to  the  OP  and  UO.  The  visual  databases  and  the  entity  models  were  also  available  to  the  multi- 
UAV  scenario. 


Ownership 

Simulation 

(STRIVE) 


Track 

Simulation 

(STRIVE) 


Figure  10:  DMS  system  architecture  and  interfaces 
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3.3  Independent  variables 

The  effectiveness  of  the  prototype  IAI  system  on  aiding  operators’  performance  was  examined 
using  a  2  (Interface  Condition:  IAI  ON  vs.  IAI  OFF)  x  3  (Operator  Workload:  Mission  Part  1  vs. 
Mission  Part  2  vs.  Mission  Part  3)  repeated  measures  design.  Thus,  each  participant  experienced 
each  of  the  six  experimental  conditions  created  by  factorially  crossing  these  two  variables. 

The  two  levels  of  Interface  Condition  were  defined  as  to  whether  the  crew  was  using  either  the 
standard  interface  (i.e.,  IAI  OFF)  or  the  standard  interface  augmented  with  the  IAI  multi-agent 
system  (i.e.,  IAI  ON).  The  IAI  for  each  participant  was  tailored  to  suit  their  needs  in  the 
following  ways.  The  UP’s  IAI  provided  an  automated  route  planning  feature  that  was  able 
to  be  activated  under  certain  conditions.  The  UO’s  IAI  would  automatically  orient  its  EO 
turret  such  that  the  closest  unknown  contact  could  be  seen.  The  TN’s  IAI  provided  a  tactical  plot 
that  indicated  how  long  it  would  take  each  UAV  to  reach  the  closest  unknown  contact. 
The  TACPLOT,  which  was  primarily  used  by  the  UP  and  the  UO,  would  automatically  re-centre 
and  re-scale. 

The  three  levels  of  Operator  Workload  were  built  into  the  three  parts  of  the  mission  scenario. 
Part  1  of  the  scenario  was  to  induce  the  lowest  workload  by  involving  in  only  one  Vertical 
Takeoff  UAV  (VTUAV)  and  only  one  contact.  Part  2  of  the  scenario  was  to  produce  moderate 
workload.  It  involved  the  control  of  two  UAVs  (i.e.,  one  VTUAV  and  one  mini  UAV)  and  the 
task  of  prosecuting  two  contacts.  Part  3  of  the  scenario  was  to  induce  the  highest  workload.  It 
involved  the  control  of  up  to  five  UAVs  (i.e.,  one  VTUAV,  three  mini  UAVs  and  one  laser 
designator  UAV)  and  the  task  of  prosecuting  three  contacts  while  keeping  “eyes-on”  a  fourth 
contact.  Part  3  of  the  scenario  had  a  higher  level  of  time  pressure  than  Parts  1  and  2  because  two 
CF-18s  (not  under  the  control  of  the  UAV  crew)  were  inbound  and  required  timely  laser 
designation  of  the  terrorist  vessel. 

3.4  Dependent  variables 

Both  objective  and  subjective  measures  were  used  to  index  each  participant’s  performance  under 
(a)  the  two  levels  of  IAI  condition  and  (b)  the  three  levels  of  operator  workload  condition.  There 
were  two  subjective  measures  for  the  experiment:  perceived  workload  and  perceived  SA.  There 
were  five  objective  measures:  completion  time  for  Critical  Task  Sequences  (CTSs),  percentage  of 
CTS  shedding,  UAV  route  trajectory  score,  UAV  airspace  violation  time,  and  Situation 
Awareness  Global  Assessment  Technique  (SAGAT)  score. 

3.4.1  Objective  measures 

Completion  Time  for  CTSs  As  a  primary  measure,  it  was  the  time  in  seconds  to  complete  all  of 
the  CTSs.  The  CTSs  were  defined  previously  using  a  Hierarchical  Goal  Analysis  (HGA) 
conducted  in  the  first  phase  of  the  project  (Hou  &  Kobierski,  2006),  but  were  limited  to  tasks  that 
had  objective  and  clearly  observable  start  and  end  points.  They  were  further  constrained  by  the 
scenario  itself.  The  tasks  had  to  be  performed  by  each  crew  in  the  same  manner  so  as  to  avoid 
excessive  variability  in  the  data  between  the  different  crews. 
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Percentage  of  CTS  Shedding  This  measure  was  to  compliment  the  CTS  completion  time 
measure.  It  highlights  deficiencies  in  the  strategy  of  quickly  and  accurately  completing  only  a 
few  critical  tasks  at  the  cost  of  not  attending  to  other  critical  tasks  under  high  workload 
conditions.  Given  that  only  successfully  completed  CTSs  were  included  in  the  completion  time 
analysis,  the  strategy  of  shedding  tasks  would  not  be  captured  in  these  data.  That  is,  crews  that 
adopted  this  strategy  would  appear  to  have  performed  well  because  their  completion  times  would 
have  been  short.  However,  their  performance  might  have  been  poor;  depending  on  the  number  of 
tasks  they  shed  (the  ones  they  did  not  complete).  The  percentage  of  CTS  shedding  measure 
highlighted  the  number  of  incomplete  tasks. 

The  percentage  of  CTS  shed  was  calculated  by  dividing  the  number  of  valid  (i.e.,  unaffected  by 
extraneous  variables)  incomplete  CTS  subsets  by  the  total  number  of  CTS  subsets  and  then 
multiplying  this  result  by  100.  Unlike  the  CTS  task  completion  time  measure,  the  CTS  shedding 
measure  was  only  calculated  for  all  CTSs  and  not  for  CTS  subset.  It  was  not  necessary  to 
describe  this  measure  in  terms  of  a  CTS  subset  as  it  was  calculated  in  terms  of  a  percentage.  It  is 
therefore  not  susceptible  to  the  statistical  issues  associated  with  (a)  an  unequal  number  of  CTSs 
across  mission  parts  and  (b)  different  tasks  across  mission  parts.  As  with  the  CTS  completion 
time  measure,  the  CTS  task  shedding  was  only  analyzed  for  Parts  2  and  3  of  the  mission  scenario. 

Airspace  Violation  Time  The  violation  time  was  the  frequency  with  which  participants  violated 
(i.e.,  flew  within)  a  half  nm  radius  surrounding  each  contact.  This  “stand-off’  zone  was  briefed 
as  a  “no-fly”  area  and  time  spent  within  this  area  was  counted  against  the  crew.  The  measure  is 
the  average  amount  of  time  (in  seconds)  that  each  crew  spent  flying  their  critical  UAVs  in  the  half 
nm  radius  that  surrounded  each  contact.  This  measure  was  to  provide  a  detailed  perspective  on 
the  crews’  navigational  errors  by  highlighting  the  temporal  magnitude  of  each  airspace  violation. 

Route  Trajectory  Score  This  measure  was  to  assess  the  effectiveness  of  the  participants  to  fly  the 
best  routes  to  investigate  all  potential  contact.  It  was  calculated  independently  for  each  critical 
UAV  in  each  of  the  three  mission  parts.  There  was  one  critical  UAV  in  Part  one,  two  critical 
UAVs  in  Part  two  and  four  critical  UAVs  in  Part  three.  These  seven  critical  UAV  trajectory 
scores  were  calculated  for  both  IAI  conditions  (ON  and  OFF),  therefore  yielding  a  total  of 
fourteen  route  trajectory  scores  for  each  crew.  The  route  trajectory  score  for  each  critical  UAV 
was  calculated  by  determining  the  average  difference  in  distance  between  the  actual  trajectory 
and  the  optimal  trajectory.  As  such,  a  lower  route  trajectory  score  indicated  better  performance. 
With  that  in  mind,  a  correction  factor  was  applied  to  each  route  trajectory  score,  which  penalized 
crews  for  violating  a  half  nm  no-fly  airspace  that  surrounded  each  contact.  This  correction  factor 
increased  the  route  trajectory  score  proportionally  with  the  amount  of  time  spent  violating 
the  half  nm  airspace. 

SAGAT  Score  Based  on  Endsley’s  (1995)  research,  SAGAT  is  a  means  to  measure  SA  when 
operators  perform  a  task.  Following  completion  of  each  of  the  six  experimental  sessions,  each 
participant  was  asked  to  re-create  their  memory  for  the  location,  heading,  and 
classification/identification  of  all  contacts  within  an  80  nm  radius  of  their  own  location  by 
plotting  them  on  a  sheet  of  8.5  inch  by  11  inch  graph  paper.  The  accuracy  of  these  plots  was 
subsequently  evaluated  by  an  SME  and  given  a  rating  of  1  through  10  according  to  a  modified 
Cooper-Harper  rating  scale,  where  a  score  of  1  indicated  excellent  SA  and  10  indicated  very  poor 
SA  (Cooper  &  Harper,  1969). 
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3.4.2  Subjective  measures 


Two  subjective  measures  were  used  to  evaluate  crew  performance,  perceived  SA  and  perceived 
workload.  Participants  were  given  a  questionnaire  at  the  end  of  each  of  the  six  sessions  that 
asked  them  to  rate  their  overall  SA  and  their  workload.  Perceived  SA  was  rated  on  a  seven-point 
Likert  scale  where  1  represented  “very  low”  SA  and  7  represented  “very  high”  SA. 
Crewmembers  were  asked  to  circle  a  number  from  one  to  seven  that  they  felt  represented  their 
average  overall  SA  for  that  session. 

The  perceived  workload  measure  was  derived  from  the  NASA  Task  Load  Index  (TLX)  Workload 
assessment  scale  (Hart  &  Staveland,  1988).  Participants  were  asked  to  rate  their  workload  on  a 
100-point  scale  for:  Mental  Demand,  Physical  Demand,  Temporal  Demand,  Frustration  Level, 
Effort,  and  Performance.  An  important  measure  derived  from  the  TLX  Workload  scale  is  overall 
workload,  which  represents  a  summary  of  these  six  workload  subscales.  One  approach  would  be 
to  average  the  six  subscale  scores  and  use  the  mean  as  each  crewmember’s  overall  workload. 
This  method,  although  straightforward,  would  be  misleading  given  that  the  relative  importance  of 
each  of  the  six  subscales  is  most  likely  unequal.  Hence,  weightings  were  assigned  to  the  six 
subscales  to  determine  their  relative  importance  using  two  different  methods:  the  pair-wise 
comparison  technique  and  the  subjective  weighting  technique.  Each  member  of  five  of  the  eight 
crews  and  all  three  NAVCOMs  involved  in  the  experiment  were  asked  to  provide  these  relative 
weighting  measures. 

The  pair-wise  comparison  technique  requested  the  participants  to  select  which  of  two  subscale 
measures  of  workload  (e.g.,  mental  demand  vs.  temporal  demand)  had  a  greater  influence  on  the 
crew  during  the  experiment.  Each  combination  of  the  six  subscales  was  compared  for  a  total  of 
fifteen  comparisons.  The  responses  were  tabulated  and  the  rankings  were  summed  to  reveal  the 
relative  contribution  of  each  workload  factor.  For  the  subjective  weighting  technique,  the  same 
respondents  were  asked  to  indicate  a  percentage  value  for  each  of  the  six  subscale  workload 
factors  that  reflected  its  relative  importance  to  the  overall  goals  of  the  mission,  where  0%  was  low 
importance  and  100%  was  high  importance.  They  were  further  instructed  to  ensure  that  these  six 
percentages  summed  to  exactly  100%.  These  percentage  values  were  then  averaged  across  all 
eighteen  respondents.  The  results  from  these  two  methods  for  determining  relative  weights  (i.e., 
pair-wise  comparisons  and  subjective  weightings)  were  then  averaged  to  provide  one  final 
weighting  value  for  each  of  the  six  subscales  of  workload.  Each  TLX  subscale  workload  score 
was  then  multiplied  by  its  final  weighting  value.  These  products  were  then  summed  to  yield  the 
overall  perceived  workload  measure. 

A  separate  questionnaire  was  also  given  to  the  NAVCOM  after  each  of  the  six  mission  segments 
where  he  or  she  rated  the  performance  of  the  each  participant  separately  and  the  performance  of 
the  crew  as  a  whole.  The  NAVCOM  was  an  essential  member  of  the  crew  in  addition  to  being 
both  a  SME  and  an  important  experimental  staff.  These  characteristics  allowed  for  a  fair, 
accurate  and  independent  assessment  of  the  crew’s  performance. 

3.5  Mission  versions  and  counterbalancing 

Each  crew  was  tested  under  both  IA1  ON  and  OFF  conditions  under  each  of  the  three  Operator 
Workload  levels.  Each  crew  experienced  each  of  the  three  mission  parts  twice  (i.e.,  ON  and 
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OFF).  In  order  to  prevent  the  influence  on  the  crews’  second  exposure  to  a  mission  part  from 
their  first  experience  with  an  identical  part,  the  second  version  of  each  part  was  created  by 
rotating  the  original  mission  layout  to  its  mirror  image.  The  track  identification  numbers  for  each 
contact  were  changed  in  order  to  increase  the  disparity  between  the  two  versions  of  the  mission. 
Part  1  always  occurred  first,  once  with  IA1  OFF  and  once  with  1A1  ON.  Part  2  always  occurred 
second,  once  with  IAI  OFF  and  once  with  IAI  ON.  Part  3  always  occurred  last,  once  with  IAI 
OFF  and  once  with  IAI  ON.  In  order  to  prevent  practice  effects  from  contaminating  the  data  (i.e., 
improved  crew  performance  due  to  increasing  familiarity  with  the  user  interface  and  UAV  flight 
dynamics),  Interface  Condition  was  balanced  across  crews.  That  is,  half  of  the  crews  received  the 
IAI  OFF  condition  first  and  the  IAI  ON  condition  second.  This  order  was  reversed  for  the  other 
half  of  the  crews.  Even  though  every  attempt  was  made  to  equate  the  original  version  of  a 
mission  part  with  its  modified  version,  there  remains  the  possibility  that  they  were  not  of  equal 
difficulty.  Mission  version  was  also  counterbalanced  across  crews.  That  is,  half  of  the  crews 
received  the  original  mission  parts  first  and  the  modified  version  second.  This  order  was 
reversed  for  the  other  crews.  Four  crews  were  required  for  a  complete  balanced  design,  as  shown 
in  Table  1. 


Table  1:  A  fully  counter-balanced  experimental  design 


IAI  Condition 

Scenario  Part 

IAI  ON 

1 

2 

3 

IAI  OFF 

1 

2 

3 

Low  Medium  High 

Level  of  Temporal  Workload 

3.6  Task  and  procedure 

The  experiment  started  with  a  ten-minute  introduction  to  participants  about  the  general  purpose  of 
the  experiment.  The  crews  were  then  given  a  two-hour  training  on  the  use  of  the  workstation 
interface.  During  this  training  session,  the  crews  were  given  a  practice  scenario.  They  were 
tasked  to  investigate  a  single  contact  using  a  VTUAV.  The  crews  practiced  until  they  were 
familiar  enough  with  their  workstation  functions,  and  they  could  identify  this  contact  both  quickly 
and  effectively. 

After  the  practice  session,  the  crews  were  given  a  twenty-minute  briefing  by  the  NAVCOM. 
NAVCOM’s  primary  role  was  to  act  as  the  liaison  between  the  experimental  staff  and  each  crew. 
The  detailed  briefing  covered  various  mission  aspects  including  the  mission  scenario  and 
the  number  of  mini  UAVs  they  had  at  their  disposal.  Most  importantly,  it  informed  the  crews 
about  the  key  visual  features  that  distinguished  the  terrorist  vessel  from  other  unknown 
vessels  (i.e.,  hydraulic  lifts  on  the  side  of  a  ship  container  and  doors  that  opened  horizontally 
instead  of  vertically). 
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The  crews  began  the  first  of  six  experimental  sessions  after  all  questions  regarding  the 
information  covered  in  this  briefing  were  answered.  The  first  experimental  session  took  about 
twenty  minutes  to  complete.  After  this  session,  each  participant  was  asked  to  complete  the 
subjective  questionnaire.  They  were  asked  to  rate  various  aspects  of  their  SA,  performance, 
confidence,  and  workload. 

The  crews  began  the  second  of  six  experimental  sessions  after  completion  of  the  first 
questionnaire.  The  second  session  also  took  about  twenty  minutes  to  complete,  and  then 
the  crews  were  again  given  a  questionnaire.  The  first  day  of  testing  ended  once  the  questionnaire 
was  completed. 

On  the  second  day,  the  crews  completed  the  remaining  four  experimental  missions  and  the 
associated  questionnaires.  The  crews  completed  two  of  the  sessions  in  the  morning  and  two  in 
the  afternoon.  The  third  and  fourth  sessions  each  took  about  twenty  five  minutes  to  complete, 
and  the  fifth  and  sixth  sessions  required  about  thirty  minutes.  The  crews  were  then  asked  to 
complete  a  usability  questionnaire  regarding  the  realism  of  the  scenario  and  the  efficacy  of  the 
operator-machine  interface  (e.g.,  the  TACPLOT,  the  PEP)  upon  completion  of  the  sixth  (and 
final)  session  and  questionnaire.  They  were  finally  debriefed  about  the  critical  aspects  of  the 
experiment.  The  crews  were  also  reminded  not  to  divulge  any  of  this  information  to  other  military 
personnel  at  CFB  Comox  or  CFB  Greenwood  as  it  could  contaminate  the  data  if  future  crews 
knew  the  purpose  of  the  experiment  prior  to  participating  in  it. 
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4  Experimental  results 


Statistical  analyses  were  performed  for  eight  dependent  measures  on  eight  crews  with  two 
completed  sets  of  counterbalanced  data.  All  data  were  subjected  to  a  paired  and  three  paired 
samples  t-test  comparing  performance  across  both  levels  of  1A1  Condition  (ON  vs.  OFF)  and 
three  levels  of  Operator  Workload  (Mission  Part  1  vs.  2  vs.  3),  respectively.  These  eight 
measures  are: 

1 .  Completion  time  for  various  CTS  (shorter  CTS  completion  time  is  better) 

2.  Percentage  of  CTS  shedding  (fewer  sequences  shed  is  better) 

3.  Corrected  route  trajectory  score  (values  <  half  nm/min  represent  a  good  trajectory  with  little 
wasted  time) 

4.  Number  of  times  that  the  UAVs  flew  within  half  nm  of  a  vessel  (lower  number  of 
times  is  better) 

5.  SAGAT  score  for  six  experimental  sessions  (higher  score  is  better) 

6.  Perceived  SA  for  various  tasks  (higher  score  is  better) 

7.  Perceived  workload  associated  with  mission  tasks  (low  ratings  are  better) 

8.  NAVCOM  assessment  of  crew  performance  (higher  ratings  are  better) 

The  results  and  associated  95%  confidence  intervals  for  both  levels  of  IAI  Condition  and  three 
levels  of  Operator  Workload  are  shown  in  Figures  11,12  and  14  to  19. 

Although  CTSs  were  determined  a  priori  for  Part  1,  they  were  not  included  in  this  analysis  as  they 
were  of  little  interest  due  to  the  minimal  workload  associated  with  this  part  of  the  scenario.  As 
such,  only  CTSs  and  other  measures  associated  with  Parts  2  and  3  will  be  discussed. 

4.1  Critical  Task  Sequence  (CTS)  completion  time 

The  CTS  data  were  subjected  to  a  2  (IAI  ON  vs.  OFF)  x  2  (Operator  Workload:  Medium  vs. 
High)  repeated  measure  of  analysis  of  variance  (ANOVA).  Although  CTSs  were  identified  for 
Part  1,  they  were,  for  the  most  part,  qualitatively  different  from  those  identified  for  Parts  2  and  3. 
Therefore,  it  is  inappropriate  to  compare  CTS  data  from  Part  1  to  Parts  2  and  3.  Consequently, 
only  CTS  data  from  Parts  2  and  3  are  analyzed  here. 

There  was  a  significant  effect  of  Workload  (Mission  Part):  where  crews  were  significantly 
faster  at  completing  CTSs  in  Mission  Part  3  (59.8  sec,  SD  =  23.6)  than  in  Mission  Part  2 
(97.9  sec,  SD  =  35.1),  F  (1,  7)  =  9.79,  MSE  =  1198,  p  <  .05.  These  data  are  shown  in  Figure  11. 
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Mean  CTS  Completion  Time  as  a  Function  of 
Operator  Workload  and  Interface  Condition 
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Figure  11:  CTS  completion  time  as  a  function  of  workload  and  interface  condition 


4.2  Percentage  of  CTS  shedding 

The  CTS  task  shedding  data  were  subjected  to  a  2  (IAI  ON  vs.  OFF)  x  2  (Operator  Workload: 
Medium  vs.  High)  repeated  measure  of  ANOVA.  Data  from  Part  1  are  not  included  in  this 
analysis  for  the  same  reason  as  stated  for  the  CTS  completion  time  data. 

There  was  a  significant  effect  of  Workload  (Mission  Part):  where  crews  shed  significantly 
more  tasks  in  Mission  Part  3  (26.8%,  SD=10.6)  than  in  Mission  Part  2  (5.5%  sec, 
SD  =  5.4),  F  (1,  7)  =  64.65,  MSE  =  54,  p  <  .001.  These  data  are  shown  in  Figure  12. 
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Percentage  of  CTSs  Shed  as  a  Function  of 
Operator  Workload  and  Interface  Condition 


Figure  12:  Percentage  of  CTS  Shed  as  a  function  of  workload  and  interface  condition 


4.3  Route  Trajectory  Score 

Figure  13  shows  an  example  of  the  Route  Trajectory  Score,  which  is  a  sample  plot  from  Crew  7 
performance  during  Part  3  of  IAI  OFF.  The  data  associated  with  the  mission  part  are  contained  in 
the  bottom  left  of  the  plot.  Each  plot  data  set  contains  information  of  the  scale  of  the  drawing  and 
the  numbers  of  the  UAV  and  boat  being  investigated  as  well  as  the  relevant  results  for  the  mission 
part.  These  results  include: 

1 .  The  time  that  the  UAV  was  flown  within  the  half  nm  stand-off  (no  fly)  circle  from  each  boat; 

2.  The  mission  time  at  which  the  UAV  first  approach  to  a  point  within  3  nms  of  the  boat.  This 
was  used  for  subsequent  calculations  of  time  to  complete  critical  task  sequences;  and 

3.  The  trajectory  score,  which  is  the  average  number  of  nms  unused  (per  minute)  because  the 
UAV  was  not  flown  at  high  speed  directly  towards  the  closest  unknown  boat. 
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18  January  2006  Crew  7  Scenario  2  Part  3d  IAI  OFF 


Figure  13:  Sample  of  trajectory >  plot 

The  corrected  route  trajectory  score  data  were  subjected  to  a  3  (Operator  Workload:  Low  vs. 
Medium  vs.  High)  x  2  (Interface  Condition:  IAI  ON  vs.  OFF)  repeated  measure  of  ANOVA. 
There  was  a  significant  effect  of  Mission  Part,  F  (2,  14)  =  37.55,  MSE  =  .02,  p  <  .001.  Three 
post-hoc  paired  samples  t-test  using  a  Bonferroni  correction  to  guard  against  alpha  slippage 
(i.e.,  dividing  the  set  alpha  level  (.05)  by  the  number  of  post  hoc  comparisons  (3)  to  obtain  a  new 
alpha  level  (0.016))  showed  that  the  crews’  route  trajectory  was  significantly  better  in 
Mission  Part  1  (0.26,  SD  =  0.18)  than  in  Mission  Part  2  (0.43,  SD  =  0.16)  or  in  Mission  Part  3 
(0.68,  SD  =  0.09),  t(7)  =  4.67,  p  <  .005,  t(7)  =  6.78,  p  <  .001,  respectively.  Further,  the  crews’ 
route  trajectory  was  significantly  better  in  Mission  Part  2  than  in  Mission  Part  3, 
t(7)  =  5.67,  p<. 001. 

The  main  effect  of  Interface  Condition  was  also  significant,  F  (1,  7)  =  21.77,  MSE  =  .02, 
p  <  .005,  with  the  crews’  route  trajectory  being  significantly  better  with  IAI  ON  (0.36,  SD  =  0.15) 
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than  with  IAI  OFF  (0.55,  SD  =  0.12).  The  interaction  between  Mission  Part  and  Interface 
Condition  was  also  significant  F  (2,  14)  =  11.71,  MSE  =  .02,  p  <  .005.  As  predicted,  this 
interaction  took  the  form  in  which  the  benefits  of  the  IAI  increased  as  operator  workload 
(scenario  complexity)  increased.  These  data  are  shown  in  Figure  14. 


Corrected  Trajectory  Score  as  a  Function  of 
^  Operator  Workload  and  Interface  Condition 


1  2  3 


Operator  Workload 


Figure  14:  Trajectory >  score  as  a  function  of  workload  and  interface  condition 


4.4  Airspace  violation 

The  number  of  times  each  crew  violated  the  'A  nm  radius  surrounding  each  contact  with  one  of 
the  critical  UAVs  was  summed  to  provide  a  cumulative  score.  The  restriction  was  that  a  UAV 
could  only  violate  the  airspace  surrounding  any  given  contact  once  for  each  mission  part.  As 
described  above,  there  were  fourteen  critical  UAV-contact  parings  for  each  crew.  Thus,  the 
maximum  number  of  airspace  violations  for  any  given  crew  is  fourteen.  These  airspace  violation 
data  were  subjected  to  a  3  (Operator  Workload:  Low  vs.  Medium  vs.  Fligh)  x  2  (Interface 
Condition:  IAI  ON  vs.  OFF)  repeated  measure  of  ANOVA.  There  was  a  significant  effect  of 
Mission  Part,  F  (2,  14)  =  10.63,  MSE  =  .38,  p  <  .005.  Three  post-hoc  paired  samples  t-test  (using 
a  Bonferroni  correction)  showed  that  the  crews  had  a  significantly  fewer  number  of 
airspace  violations  in  Mission  Part  1  (2  violations)  than  in  Mission  Part  3  (18  violations), 
t(7)  =  4.00,  p<. 005. 
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The  effect  of  Interface  Condition  was  also  significant,  F  (1,  7)  =  6.82,  MSE  =  .69,  p  <  .05,  with 
the  crews  having  significantly  fewer  airspace  violations  when  IAI  was  on  (8  violations)  than 
when  it  was  off  (23  violations).  The  interaction  between  Mission  Part  and  Interface  Condition 
was  also  significant  F  (2,  14)  =  6.66,  MSE  =  .37,  p  <  .01.  As  predicted,  this  interaction  took  the 
form  in  which  the  benefits  of  the  IAI  increased  as  operator  workload  (scenario  complexity) 
increased.  These  data  are  shown  in  Figure  15. 


Number  of  Airspace  Violations  as  a  Function 
of  Operator  Workload  and  Interface  Condition 


Figure  15:  Number  of  airspace  violations  as  a  function  of  workload  and  interface  condition 


4.5  Situation  Awareness  Global  Assessment  Technique 
(SAGAT)  score 

The  SAGAT  data  were  subjected  to  a  3  (Operator  Workload:  Low  vs.  Medium  vs.  High)  x  2 
(Interface  Condition:  IAI  ON  vs.  IAI  OFF)  repeated  measure  of  ANOVA.  The  effect  of  Interface 
Condition  was  significant,  t(7)  =  4.17,  p  <  .005,  with  crews  having  significantly  lower  SA  (higher 
SAGAT  scores)  when  IAI  was  OFF  (5.60,  SD  =  0.69)  than  when  it  was  ON  (4.82,  SD  =  0.45). 
These  data  are  shown  in  Figure  1 6. 
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SAGAT  Score  as  a  Function  of 
Operator  Workload  and  Interface  Condition 
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Figure  16:  SAGAT  score  as  a  function  of  workload  and  interface  condition 


4.6  Perceived  Situation  Awareness  (SA) 

The  perceived  SA  data  were  subjected  to  a  3  (Operator  Workload:  Low  vs.  Medium  vs.  High)  x  2 
(Interface  Condition:  IAI  ON  vs.  IAI  OFF)  repeated  measure  of  ANOVA.  There  was  a 
significant  effect  of  Mission  Part,  F  (2,  14)  =  14.98,  MSE  =  .10,  p  <  .001.  Three  post-hoc 
Wilcoxon  matched  pairs  signed-ranks  tests  (using  the  Bonferroni  correction)  showed  that  the 
crews’  perceived  SA  was  significantly  higher  in  Mission  Part  1  (5.67,  SD  =  0.37)  than  in  Mission 
Part  2  (5.38,  SD  =  0.45)  or  than  in  Mission  Part  3  (5.04,  SD  =  0.48),  T(8)  =  1,  p  <  .02,  T(8)  =  0, 
p  <  .01,  respectively. 

The  effect  of  Interface  Condition  was  also  significant,  F  (1,  7)  =  5.97,  MSE  =  .13,  p  <  .05,  with 
the  crews’  perceived  SA  being  significantly  better  with  IAI  ON  (5.49,  SD  =  0.37)  than  with  IAI 
OFF  (5.24,  SD  =  0.59).  These  data  are  shown  in  Figure  17. 
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Perceived  SA  as  a  Function  of 
Operator  Workload  and  Interface  Condition 
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Figure  17:  Perceived  SA  as  a  function  of  workload  and  interface  condition 


4.7  Perceived  workload 

The  perceived  workload  data  were  subjected  to  a  3  (Operator  Workload:  Low  vs.  Medium  vs. 
High)  x  2  (Interface  Condition:  IAI  ON  vs.  OFF)  repeated  measure  of  ANOVA.  There  was  a 
significant  main  effect  of  Mission  Part,  F  (2,  14)  =  30.31,  MSE  =  87,  p  <  .001.  Three  post-hoc 
Wilcoxon  matched  pairs  signed-ranks  tests  (using  the  Bonferroni  correction)  showed  that  the 
crews’  perceived  workload  was  significantly  lower  in  Mission  Part  1  (25.04,  SD  =  9.26)  than  in 
Mission  Part  3  (50.40,  SD  =  4.78),  T(8)  =  0,  p  <  .01.  The  crews’  perceived  workload  was  also 
significantly  lower  in  Mission  Part  2  (34.37,  SD  =  4.90)  than  in  Mission  Part  3,  T(8)  =  0,  p  <  .01. 

The  effect  of  Interface  Condition  was  also  significant,  F  (1,  7)  =  12.08,  MSE  =  87,  p  <  .01,  with 
the  crews’  perceived  workload  being  significantly  lower  with  IAI  ON  (31.94,  SD  =  14.91)  than 
with  IAI  OFF  (41.28,  SD  =  12.91).  The  interaction  between  Mission  Part  and  Interface  Condition 
was  significant  as  well,  F  (2,  14)  =  4.31,  MSE  =  42,  p  <  .05.  As  predicted,  this  interaction  took 
the  form  in  which  the  benefits  of  the  IAI  generally  increased  as  operator  workload  (scenario 
complexity)  increased.  These  data  are  shown  in  Figure  18. 
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Perceived  Workload  as  a  Function  of 
Operator  Workload  and  Interface  Condition 
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Figure  18:  Perceived  workload  as  a  function  of  workload  and  interface  condition 


4.8  NAVCOM’s  assessment  of  crew  performance 

The  NAVCOM  provided  a  rating  for  each  participant  after  each  of  the  six  mission  parts.  The 
assessment  data  were  subjected  to  a  2  (IAI  Condition:  ON  vs.  OFF)  x  3  (Operator  Workload: 
Low  vs.  Medium  vs.  High)  repeated  measure  of  ANOVA.  The  NAVCOM  reported  that  the  crew 
performed  significantly  better  when  IAI  was  ON  (6.08  combined  score)  than  when  it  was  OFF 
(5.42  combined  score),  t(7)  =  3.31,  p  <  0.05. 

The  IAI  Condition  by  Operator  Workload  interaction  was  also  significant,  F  =  6.85,  p  <  0.01.  As 
can  be  seen  in  Figure  19,  this  interaction  is  produced  by  the  IAI  functionality  significantly 
improving  performance  (as  rated  by  the  NAVCOM)  for  mission  parts  2  and  3,  but  not 
for  mission  part  1 . 
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NAVCOM's  Assessment  as  a  Function  of 
Mission  Part  and  IAI  Condition 


Figure  19:  NAVCOM’s  assessment  of  crew  performance  as  a  function  of  workload  and  IAI 

condition 


4.9  Findings 

The  experimental  results  revealed  that  participants  could  manage  tasks  faster  in  Part  3 
(see  Figure  11)  although  the  workload  was  higher  in  this  part  than  any  other  part.  This  is  because 
IAI  might  help  participants  manage  some  tasks  or  perhaps  some  tasks  were  shed  (by  participants) 
as  there  were  too  many  to  handle  in  Part  3.  Flowever,  there  were  fewer  CTSs  shed  when  IAI  was 
ON  than  OFF  (see  Figure  12)  in  Part  3,  which  means  IAI  helped  participants  manage  some  tasks. 
In  addition,  subjective  results  revealed  that  overall  SA  was  improved  when  IAI  was  ON  although 
participants  had  the  highest  workload  in  Part  3  (see  Figures  17  and  18).  The  results  were 
supported  by  the  SAGAT  scores,  which  showed  a  significant  improvement  in  SA  when  IAI  was 
ON  (see  Figure  16).  Thus,  IAI  likely  helped  participants  to  work  faster  with  improved  SA. 

Further,  the  results  of  both  objective  measures  (CTS  Completion  Time,  Percentage  of  CTS 
Shedding,  UAV  Route  Trajectory  Score,  and  UAV  Airspace  Violation  Time)  and  Workload 
subjective  measure  (see  Figures  11,  12,  14,  15,  17,  and  18)  indicate  that  performance 
was  improved  and  overall  workload  was  significantly  reduced  when  IAI  was  ON  in  both 
Parts  2  and  3.  Figures  16  and  17  depicted  that  SA  was  also  improved  when  IAI  was  ON  in  both 
Parts  2  and  3.  When  IAI  helped  out  and/or  more  tasks  were  shed,  participants  could  maintain 
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better  SA  even  though  they  had  the  highest  workload  in  Part  3.  Thus,  it  is  likely  that  they  could 
manage  mission  tasks  in  a  faster  pace  with  better  trajectory  scores  and  less  airspace  violation  time 
(see  Figures  11,  12,  14,  and  15)  with  IAI  ON.  Therefore,  the  claim  of  IAI’s  positive  impact  on 
reducing  workload  and  enhancing  situation  awareness  as  well  as  performance  were  supported. 

In  general,  the  results  indicated  that  participants  performed  more  effectively  from  both 
quantitative  and  qualitative  perspectives  when  the  IAI  was  ON.  When  IAI  was  ON,  CTSs  were 
shortened,  fewer  tasks  were  shed,  the  UAV  trajectory  scores  were  better  and  the  no-fly  areas  were 
violated  less  often  (see  Figures  11,  12,  14,  and  15).  Participants’  overall  (actual  and  perceived) 
SA  was  improved  and  overall  workload  was  reduced  as  well  (see  Figures  16,  17,  and  18).  Note 
that  many  of  the  CTSs  examined  in  the  experiment  were  previously  modeled  and  simulated  in  the 
first  phase  of  this  research.  They  were  the  sequences  performed  by  the  very  same  IAI  agent 
groups  (e.g.,  route  planning  and  inter-crew  communication)  used  in  both  the  simulation  and 
experimentation  phases.  Thus,  consistent  test  results  (e.g.,  reduced  task  completion  time  and 
operator  workload)  of  these  CTSs  in  the  experimentation  phase  validated  the  network  modeling 
results  concluded  in  the  simulation  phase.  Participants’  performance  was  improved  through  the 
use  of  an  IAI  although  they  were  working  in  a  cognitively  complex  situation. 
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5  General  discussion 


The  IAI  implemented  as  a  prototype  here  is  only  a  small  subset  of  a  more  extensive  suite  of  fully 
optimized  UAV  agent  system.  However,  the  experimental  findings  indicated  that  the  control  of  a 
dynamic  and  complex  system  such  as  multiple  disparate  UAV  control  from  an  airborne  platform 
can  be  improved  through  the  use  of  a  multi-agent  IAI  suite.  Experience  and  knowledge  were 
gained  regarding  the  design  of  IAI  agents,  the  implementation  of  synthetic  IAI  prototype 
environments,  and  the  conduct  of  the  experiments  through  the  discussions  and  observations  made 
during  the  conduct  of  the  project.  Hubbard,  et  al.  (2006)  hypothesized  that  as  the  level  of 
complexity  increases  within  these  scenarios,  the  degree  to  which  software  agents  outperform 
humans  increases. 


5.1  Technical  account  of  IAI  prototype 

First,  the  experimental  set-up  was  a  well  designed  and  developed  environment  for  exploring  the 
provision  of  assistance  to  operators.  It  has  been  excellent  for  determining  how  to  produce  an  IAI. 
As  the  project  continued,  the  list  of  potential  IAI  functions  grew  when  more  knowledge  about  IAI 
systems  was  gained.  The  actual  number  of  agents  developed  in  the  experiment  was  limited.  The 
quality  of  the  implementation  was  good  though  not  at  the  quality  of  a  production  system.  Even 
so,  the  results  found  were  very  supportive  of  the  use  of  IAI  agents  in  the  complex  environment. 
A  more  complete  set  of  agents  would  result  in  increased  crew  performance. 

Second,  the  most  effective  IAI  agents  in  the  prototype  interfaces  were:  route  planning,  route 
following,  and  inter-crew  communication  support  agents.  These  three  IAI  components  were  the 
same  agents  identified  by  the  HGA  conducted  in  the  first  phase  of  the  project. 

Third,  some  aspects  of  the  experiment  limited  the  perceived  effectiveness  of  the  IAI.  The 
conventional  user  interface  was  already  well  designed  to  be  effective  according  to  human  factors 
engineering  principles.  Participants  were  trained  on  the  conventional  interface  for  which  they  had 
developed  work  strategies,  not  on  the  IAI.  They  might  rely  on  the  original  work  strategies 
because  they  were  known  and  effective  when  the  IAI  functionality  was  selected  as  ON.  In 
addition,  the  participants  came  in  for  only  two  days.  The  novelty  of  the  conventional  interface 
would  not  have  worn  off  after  the  training  sessions.  Participants  would  rather  work  with  the 
interface  manually  than  give  up  control  to  the  IAI.  These  factors  might  have  limited  effectiveness 
of  IAI  use  though  the  experimental  results  revealed  performance  improvements. 

5.2  IAI  design  recommendations 

The  other  purpose  of  this  study  is  to  develop  guidelines  for  IAI  design.  Special  attention  was 
paid  to  this  aspect  during  all  project  phases  and  a  subset  of  the  knowledge  gained  is  outlined  here. 

5.2.1  IAI  issues 

First,  feedback  presentations  are  a  high  priority  for  interface  design  when  IAI  agents  are 
employed.  This  was  demonstrated  during  the  GUI  design  effort.  For  example,  when  the 
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requirement  for  HGA  feedback  (upwards  flow  of  information  within  the  hierarchy  of  goals)  was 
indicated  as  an  important  display  item,  the  IAI  specific  communication  message  window  was 
decided  to  be  one  of  the  IAI  features  in  interfaces,  as  introduced  in  Section  2.2. 

It  is  important  that  an  operator  interface  allows  operators  to  return  to  the  state  in  effect  before  the 
IAI  reconfigures  the  display  parameters.  In  other  words,  a  “Return  to  Previous  State”  (BACK) 
button  and  a  “Default  Settings”  button  should  be  added  to  the  IAI  display.  IAIs  must  be  designed 
to  ensure  that  they  are  not  perceived  to  take  over  control.  It  is  critical  that  the  system  informs  the 
user  of  changes  on  the  interface.  The  IAI  should  either  indicate  for  a  few  seconds  where  it  is 
going,  or  indicate  what  has  been  changed. 

The  design  of  each  IAI  agent  in  a  prototype  interface  must  be  based  on  available  information  in 
real  operations.  The  interface  designer  must  be  realistic  with  regards  to  the  information  available 
on  a  system  data  inventory.  In  a  modelling  and  simulation  environment,  the  design  engineers 
know  the  ground  truth.  However,  some  information  used  in  simulation  cannot  be  obtained  from  a 
real  live  subsystem.  In  other  words,  during  the  design  and  development  of  an  IAI,  every  attempt 
should  be  made  to  ensure  that  an  IAI  agent  “is  aware”  of  the  state  of  the  world.  This  may  include 
access  to  data  fusion  interim  variables  and  associated  probabilities,  which  would  allow  the  IAI  to 
produce  strategies  that  “play  the  odds”. 

There  is  a  qualitative  difference  that  an  IAI  interface  requires  an  operator  to  manage  his  or  her 
UAVs.  Using  a  conventional  interface  the  operator  provides  heading  and  altitudes.  When  an  IAI 
is  used,  the  operator  provides  goals  or  mission  objectives.  The  IAI  must  be  designed  to  not 
misinterpret  intentions,  for  example,  initiating  a  holding  pattern  when  a  pilot  is  trying  to  fly  to  a 
distant  refuelling  location.  The  operator  moves  beyond  inserting  parameters  but  rather  insert  or 
establish  system  objectives.  The  design  and  development  team  should  evaluate  the  operator’s 
activities  and  try  to  move  his/her  thinking  from  the  operating  level  to  the  strategic  level. 

All  IAI  functions  should  be  studied  during  design  and  development  and  incoiporated  into  a  user 
evaluation.  A  thorough  investigation  also  should  be  conducted  to  confirm  that  the  concept  is 
implemented  adequately  for  the  assessment.  Poor  implementation  will  not  increase  the  value  of 
any  IAI  function  but  mask  the  true  acceptance  or  potential  improvement  in  effectiveness. 

5.2.2  Potential  IAI  roles 

There  were  only  six  sets  of  IAI  agents  (functional  groups)  implemented  in  the  prototype 
interfaces.  There  are  other  sets  of  IAI  candidates  which  are  good  at  assisting  operators  in 
effective  decision-making,  especially  in  the  context  of  multiple  UAV  control. 

First,  an  IAI  agent  would  be  able  to  generate  suggestions.  For  example,  when  a  UAV  is  flying 
away  for  the  perceived  scene  of  action,  the  agent  may  query  an  operator  with  a  question  similar 
to:  “Do  you  want  me  to  turn  UAV  3  around?”  The  agent  would  have  to  recognize  that  a  UAV 
was  being  ignored  or  had  been  forgotten. 

Second,  an  IAI  agent  may  build  on  other  existing  software  to  help  complete  part  of  the 
background  data  analysis.  For  example,  the  data  fusion  calculations  based  on  different  routes  of 
different  UAVs  could  lead  to  optimal  route  selection  to  maximize  fused  data.  This  information 
would  be  used  by  an  IAI  route  planning  agent. 
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Third,  an  IAI  agent  may  intelligently  provide  information  to  the  crew  by  sequentially  selecting 
the  most  relevant  data  items  for  display.  This  could  be  similar  to  the  CNN  news,  which 
continuously  moves  from  a  news  story  to  another  news  story  with  the  sequence  depending  on  the 
latest  breaking  information  and  the  relevance  to  the  viewers.  In  this  way,  a  UAV  operator  could 
adjust  to  the  most  recent  video  from  the  team  of  UAVs  investigating  contacts.  This  may  lead  to  a 
“ticker  tape”  of  information  scrolling  across  the  display. 

Fourth,  an  IAI  agent  may  produce  a  “heading”  path  for  every  UAV  to  indicate  (to  an  operator) 
where  each  UAV  will  be  flying.  The  “heading”  path  would  be  reassessed  and  modified  as 
required.  If  a  contact  were  to  “pop-up”,  the  agent  would  plan  routes  to  accommodate  the 
unknown.  The  agent  could  produce  a  plan  and  offer  it  up  for  acceptance,  or  it  could  implement 
the  new  plan  and  modify  the  UAV  paths  shown  on  the  TACPLOT.  Each  optimal  flight  path  may 
have  holding  patterns  identified.  A  design  challenge  would  be  the  level  of  autonomy  allocated  to 
the  group  of  UAVs  and  identification  of  the  appropriate  holding  points  at  which  operators’ 
overview  would,  should,  or  must  occur. 

Fifth,  an  IAI  agent  may  be  self-preservative.  This  IAI  function  would  maintain  altitude 
separation  between  UAVs  and  would  warn  a  pilot  if  a  UAV  was  descending  into  the  water. 
Additionally,  the  agent  would  ensure  that  the  engine  was  operating  and  a  minimal  floor  height 
maintained  (with  a  standard  holding  pattern  if  necessary)  to  keep  the  UAV  from  crashing. 

Sixth,  another  IAI  agent  may  assess  the  utility  of  all  of  the  UAVs.  Each  UAV  would  be  assessed 
for  utility  and  possible  redeployment.  This  would  identify  UAVs  that  were  flying  away  from  the 
area  of  interest.  This  is  a  case  where  the  agent  could  suggest  a  new  route  (or  a  holding  pattern) 
and  ask  an  operator  if  he  or  she  would  like  to  have  the  new  route  implemented.  The  agent  could 
also  advice  the  operator  to  automatically  redirect  the  UAV  to  prosecute  an  unknown  contact. 
Similarly,  the  agent  could  be  controlled  to  prosecute  multiple  targets  efficiently.  The  agent  could 
implement  a  minimum  cost  function  to  use  the  available  resources  to  investigate  all  contacts  as 
quickly  as  possible. 

Finally,  each  IAI  agent  may  have  a  different  level  of  autonomy  appropriate  to  the  function.  An 
example  of  various  levels  of  autonomy  would  be  a  pilot  accepting  full  route  planning  and  route 
following  while  a  sensor  operator  uses  an  IAI  “advise”  mode  rather  than  full  turret  repositioning. 
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6  Conclusions 


This  research  assessed  the  efficacy  of  IAI  technology  in  a  dynamic  and  complex  military  system, 
and  further  provided  generic  design  guidance  on  IAI  systems.  A  task  network  model  was 
developed  based  on  a  HGA  for  an  operational  scenario  which  involved  controlling  multiple 
UAVs  from  an  airborne  platform.  Three  IAI  functional  agent  groups  were  simulated  in  a 
performance  modeling  environment  to  predict  operators’  performance.  The  same  three  agents 
and  other  three  IAI  function  groups  were  implemented  in  a  synthetic  prototype  system  for  an 
empirical  evaluation.  The  experimental  results  confirmed  the  findings  of  performance  modeling 
and  further  supported  the  hypotheses  about  using  IAIs  to  reduce  workload  and  improve  SA  and 
performance.  Although  IAIs  were  operated  in  either  ON  or  OFF  mode  in  the  prototype 
interfaces,  the  research  findings  showed  potential  optimal  conditions  to  trigger  IAIs  under 
different  cognitive  workload  situations. 

This  study  is  a  typical  example  of  designing  an  IAI  as  an  augmented  cognition  aid  to  address 
complexity,  dynamics,  and  information  overflow  issues  associated  with  many  military  systems 
used  in  network  centric  warfare  or  any  other  networked  systems.  The  benefits  of  using  IAIs  in 
these  systems  for  the  intelligent  adaptive  assisting  of  the  decision-making  processes  have  been 
demonstrated.  Knowledge  obtained  from  the  processes  of  concept  development,  performance 
modeling,  prototype  implementation,  and  experimentation  has  helped  to  generate  preliminary 
guidelines  for  designing  IAI  systems. 
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Annex  A  Experimental  protocol 


Original  DRDC  Toronto  Protocol  #L-5 1 1 

Title:  Experimental  Assessment  of  Intelligent  Adaptive  Interfaces  for  UAV  Operators 
Principle  Investigator:  Dr.  Ming  Hou,  DRDC  Toronto  (Thrust  13il) 

Run  Director:  Mr.  Bob  Kobierski,  CMC  Electronics  Inc.  (Ottawa) 


A.1  Executive  Summary 

A.1.1  Background 

The  Canadian  Forces  (CF)  is  considering  more  widespread  use  of  Uninhabited  Air  Vehicles 
(UAVs)  to  provide  a  new  integrated  intelligence,  surveillance,  and  reconnaissance  (IISR) 
capability.  In  this  role,  UAVs  would  be  force  multipliers,  releasing  manned  aircraft  for  other 
roles.  Elowever,  at  the  moment  UAV  control  is  operator  intensive  and  can  involve  high  levels  of 
workload.  Intelligent  Adaptive  Interfaces  (IAIs)  are  a  potential  technology  for  controlling 
operator  workload  and  improving  decision  effectiveness  in  the  employment  and  operation  of 
these  platforms.  Defence  Research  &  Development  Canada  (DRDC)  has  initiated  a  project  for 
the  definition  and  development  of  IAIs  for  advanced  UAV  control  under  Thrust  13i  for  the  Air 
Environment.  The  aim  of  this  project  is  to  develop,  demonstrate,  and  prioritize  enabling 
technologies  that  can  be  applied  to  an  advanced  operator  interface  that  will  support  reduced 
manning  and  enhanced  performance  in  complex  military  systems,  specifically  UAV  control  from 
an  airborne  platform.  The  overall  project  involves  three  phases  and  lasts  three  years.  In  the  first 
two  phases,  theoretical  frameworks  and  design  concepts  have  been  developed,  and  followed  by 
prototype  interface  design  and  implementation.  The  work  being  conducted  in  the  last  phase  here 
is  to  conduct  experimental  evaluation  on  these  IAI  interfaces  in  CMC  Electronics  Inc  who  is 
under  contract  with  DND  to  perform  this  work. 

A.1. 2  Purpose 

The  purpose  of  this  research  is  to  conduct  an  experimental  evaluation  of  the  efficacy  of  prototype 
IAIs  in  a  simulated  environment  for  the  control  of  multiple  UAVs  from  an  airborne  platform. 

A.1. 3  Procedure 

Participants  will  be  tested  as  a  team  of  three-aircrew  individuals  in  a  simulated  UAV  control 
compartment  on  CP  140  Aurora.  The  participants  will  be  required  to  sit  in  front  of  the  simulated 
UAV  control  stations  and  work  with  their  own  operator  interfaces  in  which  a  counter-terrorist 
scenario  will  be  presented.  The  participants’  task  will  be  to  control  multiple  UAVs  and  to  search 
and  locate  a  terrorist  boat.  The  entire  experiment  will  take  about  fifteen  (15)  hours  including 
briefing,  training,  two  separate  one-hour  trials  and  one-hour  debriefing  session. 
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A.1.4  Benefit 


The  results  from  this  study  will  increase  the  understanding  of  the  impact  of  automated  systems  on 
the  enhancement  of  CP  140  capability,  especially  the  integration  with  multiple  UAV  control 
systems  on  board. 

A.1.5  Risks 

No  unusual  risks  are.  Although  there  are  no  anticipated  risks,  the  participants  will  be  informed 
that  they  have  the  right  to  terminate  the  trial  at  anytime.  The  participants  will  not  be  subjected  to 
medical  screening.  The  participants  will  not  receive  undue  physical  and/or  mental  stress  due  to 
participation  within  the  study.  No  physician  coverage  is  required  for  the  study. 


A.2  Glossary  Of  Terms  and  Acronyms 


ALIX 

AOO 

AS 

ASO 

CC-130 

CC-144 

CCG 

CF 

CFEC 

CMC 

CP  140 

CPF 

CTS 

DFO 

DRDC 

FY 

GUI 

HGA 

IAI 

II  SR 

IUI 

MALE 

NavCom 

NASO 

OGD 

OTH 

PCT 

PDT 

PEP 

PLIX 

R&D 

SAGAT 


Atlantic  Littoral  Intelligence,  Surveillance,  and  Reconnaissance  Experiment 

Area  of  Operations 

Adaptive  Systems 

Acoustic  Sensor  Operator 

Hercules 

Challenger 

Canadian  Coast  Guard 
Canadian  Forces 

Canadian  Forces  Experimentation  Centre 
CMC  Electronics  Inc. 

Aurora 

Canadian  Patrol  Frigate 
Critical  Task  Sequence 
Department  of  Fisheries  and  Oceans 
Defence  Research  &  Development  Canada 
Fiscal  Year 

Graphical  User  Interface 
Hierarchical  Goal  Analysis 
Intelligent  Adaptive  Interface 

Integrated  Intelligence,  Surveillance,  and  Reconnaissance 

Intelligent  User  Interface 

Medium  Altitude  Long  Endurance 

Navigator  Communicator 

Non-Acoustic  Sensor  Operator 

Other  Government  Department 

Over  The  Horizon 

Perceptual  Control  Theory. 

Perceptual  Detection  Task 
Programmable  Entry  Panel 

Pacific  Littoral  Intelligence,  Surveillance  and  Reconnaissance  Experiment 

Research  and  Development 

Situation  Awareness  Global  Assessment  Technique 
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SME  Subject  Matter  Expert 

TN  Tactical  Navigator 

UAV  Uninhabited  Aerial  Vehicle 

UCAV  Uninhabited  Combat  Aerial  Vehicle 

UO  UAV  Sensor  Operator 

UP  UAV  Pilot 

VTUAV  Vertical  Take-Off  UAV 

A.3  Ethics  Committee  Protocol 

A.3.1  Protocol  Number 

The  protocol  number  for  the  present  study  is:  L-5 1 1 . 

A.3. 2  Title 

Experimental  Assessment  of  Intelligent  Adaptive  Interfaces  for  UAV  Operators. 

A.3. 3  Investigator 

Principal  Investigator:  Dr.  Ming  Hou,  DRDC  Toronto 

A.3.4  Background 

The  Canadian  Forces  (CF)  is  considering  more  widespread  use  of  Uninhabited  Air  Vehicles 
(UAVs)  to  provide  a  new  Integrated  Intelligence,  Surveillance,  and  Reconnaissance  (IISR) 
capability.  In  this  role,  UAVs  would  be  force  multipliers,  releasing  manned  aircraft  for  other 
roles.  However ,  at  the  moment  UAV  control  is  operator  intensive  and  can  involve  high 
levels  of  workload.  Intelligent  Adaptive  Interfaces  are  a  potential  technology  for  controlling 
operator  workload  and  improving  decision  effectiveness  in  the  employment  and  operation  of 
these  platforms. 

Defence  Research  &  Development  Canada  (DRDC)  has  initiated  a  project  for  the  definition  and 
development  of  IAIs  for  advanced  UAV  control  under  Thrust  13i  for  the  Air  Environment.  The 
aim  of  this  project  is  to  develop,  demonstrate,  and  prioritize  enabling  technologies  that  can  be 
applied  to  an  advanced  operator  interface  that  will  support  reduced  manning  and  enhanced 
performance  in  complex  military  systems,  specifically  UAV  control  from  an  airborne  platform. 

The  overall  project  involves  the  following  three  phases  and  will  last  3  years: 

Y ear  1  Concept  development  and  theoretical  frameworks.  Y ear  1  work  is  now  complete. 

Year  2  Prototype  Interface  design  and  implementation;  including  the  creation  of  the  simulation 
environment  and  performance  measurement  suite  for  experimental  evaluation.  Y ear  2  goals  are 
now  achieved. 
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Y ear  3  Demonstration  of  the  concept  and  experimental  evaluation  including  the  production  of 
draft  design  guidelines  for  lAIs  and  the  generation  of  a  follow-on  Technology  Demonstration 
proposal.  This  phase  provides  the  scope  for  the  current  work. 

The  domain  of  application  for  the  demonstration  of  1A1  capability  is  advanced  UAV  control, 
focusing  on  those  technologies  that  increase  the  ratio  of  UAV  to  operator.  An  operational 
mission  scenario  has  been  devised,  and  hierarchical  goal  analyses/performance  modelling  was 
conducted  in  Year  1  of  the  project. 

Year  2  of  the  Project  included  the  development  of  interface  design  concepts,  the  implementation 
of  prototype  interfaces  for  the  creation  of  simulation  environments,  and  preparation  of  a 
performance  measurement  suite  for  the  future  experimental  evaluation.  The  work  of  this  phase 
also  resulted  in  an  experimental  environment  including  the  control  of  interface  intelligence, 
temporal  workload,  and  task  complexity  for  three  UAV  operators  in  order  to  run  the  experiment. 

The  work  described  hereby  is  for  Year  3  of  the  Project  and  includes  the  modification  of  both 
conventional  and  “Wizard-of-Oz”  prototype  interfaces  for  experimentation  purpose.  The  use  of 
actual  1AI  agent  software  within  the  interface  will  also  be  demonstrated.  The  experimental  data 
will  be  collected  and  analyzed  statistically  to  testify  the  efficacy  of  1AI  technology.  The 
experimental  results  will  result  in  a  preliminary  guideline  for  designing  IAIs  according  to 
the  recommendations  based  on  the  comparison  of  UAV  operator  interfaces  with  and 
without  IAI  components. 

A.3.5  Purpose  of  Study 

The  objective  of  this  experiment  is  to  conduct  an  experimental  evaluation  of  the  efficacy  of 
prototype  intelligent  adaptive  interfaces  (IAIs)  for  the  control  of  multiple  UAVs  from  an  airborne 
platform.  Based  on  the  results,  recommendations  and  guidelines  will  be  proposed  for  the  design 
of  IAIs. 

A.3.6  Selection  of  Human  Subjects 

There  will  be  8  teams  of  3  aircrew  individuals  with  age  range  from  16  to  60  recruited  as 
volunteers  from  CP  140  community  at  CFB  Greenwood  and  Comox  through  1  Canadian  Air 
Division  for  the  study.  These  27  subjects  will  have  no  prior  UAV  controlling  history  but  CP  140 
operational  exposure.  Each  team  will  consist  of:  1  UAV  pilot  (UP),  1  UAV  sensor  operator 
(UO),  and  1  Tactical  Navigator  (TN).  Preferably,  the  UAV  pilot  will  have  some  level  of 
experience  of  piloting  CP  140,  providing  that  he  or  she  has  completed  the  Operational  Training 
Unit.  Similarly,  the  UAV  operator  may  be  junior  Non-Acoustic  Sensor  Operator  (NASO), 
although  at  least  one-year  experience  would  be  highly  desirable.  A  qualified  TN  is  also  highly 
desirable,  although  an  "A"  category  Navigational  Communicator  (NAVCOM)  would  also  be 
satisfactory.  Since  all  subjects  are  well-trained  aircrews  and  tasked  to  participate  in  the 
experiment,  they  will  not  be  compensated  with  any  stress  allowance.  All  subjects  will  be  fully 
briefed  as  to  the  purpose,  details,  discomforts,  and  risks  associated  with  the  experimental  protocol 
before  being  asked  for  their  written  informed  consent.  All  subjects  will  be  required  to  sign  the 
consent  form  (see  attached  copy)  before  participating  in  the  study  to  indicate  that  they  have  been 
briefed  to  their  satisfaction  and  have  understood  the  risks. 
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A.3.7  Methodology 


Based  on  prior  hierarchical  goal  analysis  and  performance  modelling  for  UAV  control  activities, 
the  hypotheses  were  generated  for  the  experimental  evaluation.  They  are: 

■  the  IAI(s)  will  enhance  the  Situation  Awareness  and  performance  of  UAV  operators 
while  potentially  decreasing  workload. 

■  the  impact  of  lAI(s)  will  vary  across  the  three  UAV  operator  roles  (i.e.,  TN,  UO,  UP) 
and  presumably  be  greatest  for  the  most  complex  activity. 

■  the  IAI(s)  will  have  the  greatest  positive  impact  in  high  workload  situations. 
Experimental  Design 

In  order  to  test  the  hypotheses  above,  the  impact  of  the  prototype  IAIs  will  be  examined  using 
two  independent  (manipulated)  variables:  operator  workload  and  interface  condition.  Hence,  the 
experiment  has  a  3  x  2  mixed  factors  design  that  includes  two  within-subject  variables  (Operator 
Workload:  3  levels  that  correspond  to  3  mission  parts),  and  (Interface  Condition:  2  levels  that 
correspond  to  IAI  on  versus  IAI  off).  There  will  be  8  crews  of  3  team  members  (totally  24 
participants)  recruited  for  the  experiment. 

Operator  Workload 

The  missions  for  this  experiment  have  been  designed  to  incorporate  3  levels  of  workload  that 
correspond  to  the  3  mission  parts:  Par  1  Part  2,  and  Part  3.  The  fist  part  requires  the  crew  to  use 
one  UAV  to  investigate  one  contact  and  monitor  the  balance  of  the  CP  140  crew  (pilots  and 
NASOs)  investigate  a  second  contact.  During  the  second  part  of  the  mission,  the  subject  crew 
will  be  required  to  employ  two  UAVs  simultaneously,  each  investigating  a  different  contact. 
During  the  third  and  highest  workload  portion  of  the  mission  the  subject  crew  will  be  required  to 
employ  up  to  5  UAVs  at  once  while  prosecuting  a  terrorist  vessel. 

Interface  Condition 

UAV  operator  interfaces  are  computer  displays  on  UAV  control  stations.  An  operator  interface  is 
the  only  media  that  operators  can  interact  with  the  system  to  control  the  UAVs  remotely.  In  this 
study,  the  interfaces  are  graphic  user  interfaces  which  have  been  integrated  with  simulated  UAV 
control  stations  for  the  purpose  of  evaluation  on  IAI  technology. 

There  are  two  levels  of  interface  condition  correspond  to  whether  an  IAI  is  on  (IAI  ON)  or  off 
(IAI  OFF).  (Interface  condition  is  not  shown  in  Table  1).  When  IAI  is  on,  there  will  be 
automation  features  of  the  interface  assisting  operators  to  control  multiple  UAVs.  When  IAI  is 
off,  operators  will  have  to  manually  manipulate  with  the  interface  to  control  UAVs. 

Two  approximately  60-minute  missions  are  required  for  this  experiment.  These  missions  are 
designed  to  be  similar,  but  different  enough  to  counter  possible  familiarity  effects.  Two  different 
missions  are  required  because  each  crew  is  required  to  undertake  the  role-playing  exercise  with 
and  without  the  IAI  selected  ON.  For  example,  if  the  first  subject  crew  is  required  to  accomplish 
the  counter-terrorist  mission  with  the  IAI  selected  OFF  (using  the  conventional  interface  only); 
then,  when  they  are  testing  the  system  with  IAI  selected  ON,  this  crew  must  be  presented  with  a 
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scenario  with  the  unknown  boats  and  the  terrorist  boat  positioned  in  different  locations.  If  not  the 
crew  will  demonstrate  increased  performance,  not  because  the  interface  is  better,  but  because  they 
know  the  ground  truth.  In  order  to  avoid  an  effect  caused  by  the  order  in  which  the  two  missions 
are  undertaken,  the  experiment  needs  to  be  balanced  by  requiring  Mission  1  to  be  “flown”  first 
with  one  crew  and  Mission  2  to  be  “flown”  first  with  the  next  crew. 

Each  60-minute  mission  is  divided  into  3  parts,  referred  to  as  Parts  1,  2,  and  3.  Thus,  there  are 
combinations  of  Mission  (1  or  2)  and  Parts  (1,  2,  and  3).  Accordingly,  Ml-Pl  refers  to  Part  1  of 
Mission  1.  M2-P1  refers  to  Part  1  of  Mission  2,  etc.  Parts  are  always  run  in  “1-2-3”  order. 
However,  for  each  crew,  a  part  is  repeated.  For  example,  the  order  for  Crew  1  will  be  Ml-Pl 
(with  IAI  OFF),  M2 -PI  (with  IAI  ON).  This  will  allow  for  a  direct  comparison  of  the  IAI  OFF 
vs.  IAI  ON  condition  for  the  type  of  tasks  performed  in  the  early  (e.g.,  Part  1) 
components  of  the  missions. 

Each  crew  will  complete  2  full  missions,  each  with  3  parts.  For  counterbalancing  purposes, 
multiples  of  4  crews  will  be  required.  That  is,  either  4  crews,  8  crews  or  12  crews.  As  indicated 
earlier,  the  order  in  which  the  mission  and  “flown”  must  be  balanced,  but  also  the  order  in  which 
the  IAI  is  selected  to  OFF  or  ON  must  also  be  balanced.  As  a  result,  the  experiment  will  require 
one  crew  to  first  fly  with  IAI  OFF  on  Mission  One  which  will  be  balanced  with  a  second  crew 
who  fist  fly  with  IAI  ON  on  Mission  One.  These  two  crews  will  be  balance  with  two  follow-up 
crews  who  fly  Mission  Two  first.  This  is  shown  in  Table  A-l. 

Table  A- 1:  Experimental  Design:  2  (Mission)  x  2  (IAI  Mode) 


IAI  Condition 

Crew  1 

IAI  OFF 

1.  Ml-Pl 

3.  M1-P2 

5.  M1-P3 

IAI  ON 

2.  M2-P1 

4.  M2-P2 

6.  M2-P3 

IAI  Condition 

Crew  2 

IAI  ON 

1.  Ml-Pl 

3.  M1-P2 

5.  M1-P3 

IAI  OFF 

2.  M2 -PI 

4.  M2-P2 

6.  M2-P3 

IAI  Condition 

Crew  3 

IAI  OFF 

1.  M2 -PI 

3.  M2-P2 

5.  M2-P3 

IAI  ON 

2.  Ml-Pl 

4.  M1-P2 

6.  M1-P3 

IAI  Condition 

Crew  4 

IAI  ON 

1.  M2 -PI 

3.  M2-P2 

5.  M2-P3 

IAI  OFF 

2.  Ml-Pl 

4.  M1-P2 

6.  M1-P3 

A.3.8  Experimental  Environment 

The  physical  layout  of  the  experimental  environment  is  sketched  in  Figure  A-l.  Each  of  the  UAV 
crew  member  has  his  own  control  station  and  interface.  Figure  A-l  shows  the  images  of  the 
physical  layout  of  these  workstations  and  interfaces. 


44 


DRDC  Toronto  TR  2006-292 


Figure  A- 1 :  Physical  layout  of  the  experimental  environment 


The  STRIVE  (manufactured  by  CAE)  software  was  modified  to  support  the  concurrent  control  of 
multiple  UAVs.  It  provides  the  simulations  for  all  the  UAVs  and  their  sensor  payloads.  The 
visual  databases  and  the  entity  models  were  also  available  to  the  multi-UAV  scenario.  The 
followings  are  some  of  the  features  of  the  simulation  environment. 
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Video  capture 


Five  video  cameras  will  be  positioned  to  capture  critical  crew  actions  and  interactions, 
these  include: 

1.  one  camera  set  to  capture  hand  movements  of  TN,  UP,  and  UO; 

2.  as  a  minimum,  one  other  camera  to  capture  all  crew  interactions; 

3.  one  camera  to  capture  mission  time  recorded  at  5  Hz  (STRIVE  time  will  be  synchronized 
with  mission  time); 

4.  five  feeds  from  subject  displays  (four  displays  and  one  STRIVE  image)  will  be  streamed  to  a 
video-scan  conversion  unit  for  recording;  and 

5.  live  video/audio  data  may  be  projected  in  a  viewing  area  outside  the  simulation  room  to  show 
images  from  video  display  console. 

Audio  capture 

Intercom  audio  from  both  intercom  nets  must  be  recorded  with  audio  and  the  video  described 
above  captured  on  same  recorder,  thus  the  audio  will  be  time  stamped  as  per  video  capture.  An 
ambient  microphone  will  be  provided  at  all  workstations.  In  addition,  speakers  or  headsets  will 
be  located  at  experimenter  station  to  play  crew  audio. 

Mission  time  stamps 

In  accordance  with  the  video  capture,  mission  time  will  correspond  to  the  time  recorded  on  digital 
video  recorder  to  allow  collection  of  start  and  stop  times  for  segments  and  critical  task  sequences. 
A  mission  clock  will  be  provided,  which  will  be  set  to  18:00  hrs  at  start  of  Part  1  of  each  mission, 
18:20  hrs  at  start  of  Part  2,  18:40  hrs  at  start  of  Part  3. 

A.3.9  Task  and  Procedure 

The  general  task  for  the  UAV  crew  is  to  run  the  scenario  for  searching  terrorist  vessel  as  quickly 
and  accurately  as  possible.  The  experimental  procedures  will  involve: 

Day  one: 

a.  welcoming  the  subject  crew  and  introducing  them  to  the  experimental  environment; 

b.  10  minutes  coffee  break; 

c.  conduct  of  a  training  session  for  a  set  period  of  time  (2  hours); 

d.  lunch 
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e.  conduct  of  one  mission,  in  three  parts,  with  a  qualitative  data  collection  session 
following  each  part  of  the  scenario  and  a  1 0  minute  intermission; 

Day  two: 

f.  Welcoming  (crew  arrives) 

g.  1 5  minutes  coffee  break; 

h.  conduct  of  the  second  mission,  using  the  same  procedure  as  the  first  mission;  and 

i.  collection  of  qualitative  comments. 

A.3.10  Measurements 

The  main  performance  parameters  include  mission/segment  completion  time,  response  time  for 
Critical  Task  Sequences  (CTSs),  and  completion  time  for  CTSs.  CTS  are  those  tasks  identified 
by  the  previous  hierarchical  goal  analyses  as  critical  activities  to  take  to  achieve  the  overall 
counter-terrorist  goal. 

Hence,  there  are  three  objective  performance  and  effectiveness  measurements: 

1.  mission/segment  completion  time:  the  time  subjects  spend  to  complete  the  whole 
mission  or  any  one  of  the  segments, 

2.  critical  task  response  time:  the  time  subjects  realize  an  critical  event  needs  to  react 
after  it  occurs,  and 

3.  critical  task  completion  time:  the  time  subjects  spend  to  finish  a  critical  task 
once  it  is  initiated. 

Besides  above  objective  measurements,  there  are  two  other  subjective  measurements:  situational 
awareness  and  workload. 

Situational  Awareness:  Subjective  ratings  of  Situation  Awareness  (SA)  are  linked  to  the  previous 
hierarchical  goals  of  the  crew  member.  Measures  of  SA  uses  a  Situation  Awareness  Global 
Assessment  Technique  (SAGAT)  technique  at  the  end  of  each  mission  part.  Using  the  SAGAT, 
the  simulation  will  be  frozen  and  the  crew  displays  blanked.  The  UAV  crew  are  then  required  to 
provide  information  regarding  the  current  status  of  the  various  agents  in  the  mission  (e.g.,  status 
of  a  UAV  asset,  location  and  disposition  of  contacts)  as  well  as  information  regarding  the 
predicted  future  status  of  the  agents  (e.g.,  where  the  contact  will  be  in  1,  5  10  minutes).  The 
rating  form  is  attached  in  Annex  F. 

Workload:  Subjective  ratings  in  the  form  of  a  modified  NASA  TLX  will  be  used  for  assessing 
workload  at  the  end  of  the  experiment.  The  rating  criteria  are  linked  to  the  hierarchical  goals  of 
the  UAV  crew  members.  The  rating  form  is  attached  in  Annex  F. 
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A.3.11  Data  Collection 


The  data  collection  utility  was  developed  to  meet  the  specific  experimental  needs  of  this  project. 
It  is  a  relatively  small  software  package  that  taps  into  available  data  from  the  scenario 
management  tool  (STRIVE)  and  specific  data  elements  that  are  exported  by  the  Graphic  User 
Interface  (GUI)  software.  It  will  name  each  data  file  based  on  crew  number,  mission  number,  part 
number,  and  IAI  condition.  In  addition,  automatically  generated  mission  time  records  will  be 
provided.  The  data  files  will  include: 

-  mission  time; 

-  operator  (1=TN,  2=UP,  3=UO); 

-  keystroke  (string)  and  PEP  selections;  and 

-  relevant  scenario  events  initiated  by  experimentation  staff. 

The  experimenter  will  be  provided  with  an  Excel-like  spreadsheet  (on  a  computer)  for  entry  of 
start  and  stop  times  for  each  predetermined  CTS.  Mission  times  will  be  entered  automatically 
when  the  experimenter  selects  the  CTS  label. 

The  STRIVE  entity  identification  and  location  be  recorded  and  time  stamped.  This  will  provide 
an  ability  to  look  at  recorded  site  picture  at  any  mission  time.  This  entity  record  capability  is 
required  primarily  when  simulation  has  been  paused.  The  status  and  crew’s  deployment  and  use 
of  assets  (e.g.,  UAV  launch,  sensor  selections)  will  also  be  recorded. 

A.3.12  Medical  Screening 

The  subjects  used  for  this  experiment  will  be  drawn  from  the  CP140  user  community  (TN, 
NASO,  Pilot  and  NAVCOM)  and  these  members,  who  are  “fit  to  fly”,  are  considered  medially 
suitable  for  operator  interface  experiments  in  this  simulated  CP  140  tactical  compartment.  As  a 
result,  medical  screening  is  not  required. 

A.3.13  Physician  Coverage 

The  participants  will  not  receive  undue  physical  and/or  mental  stress  due  to  participation  within 
the  study.  No  physician  coverage  is  required  for  the  study. 

A.3.14  Roles  and  Qualifications  of  Team  Members 

The  Test  Team  will  be  comprised  of  employees  of  CMC  Electronics  Human  Factors  Engineering 
Team  and  Carleton  University  Aviation  and  Cognitive  Engineering  (ACE)  Laboratory.  The  Test 
Team  will  play  the  roles  of  STRIVE  Operator  (simulation  setup  and  operation)  and  data 
collection.  The  CP  140  subjects  will  operate  as  a  crew  and  will  be  supplemented  by  a  Test  Team 
member  who  will  play  the  role  of  NAVCOM.  This  person  will  be  a  serving  member  of  the 
Maritime  Proving  and  Evaluation  Unit  at  CEB  Greenwood. 

The  subject  crew  will  be  a  trained  coastal  patrol  crew  and  will  respond  to  the  pre-flight  anti 
terrorist  mission  tasking.  During  the  evolution  of  the  experiment,  the  NAVCOM  (who  is  a 
member  of  the  experimental  staff)  will  stimulate  the  subject  crew  will  “taskings”  from  the 
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Operations  Centre  (MOC)  at  Maritime  Forces  Atlantic,  in  Flalifax;  and  messages  from  other  units 
such  as  the  section  of  CF-1 8  fighters  who  join  the  scenario  in  the  third  part  of  the  mission. 


A.3.15  Risks  and  Benefits 

Risks 

This  experiment  is  judged  to  be  a  minimal  risk  endeavor  as  the  entire  mission  is  “flown”  in  a 
synthetic  environment  located  within  the  DND  HEART  TIES  Laboratory  in  Ottawa.  In  that  there 
are  no  risks  associated  with  the  experiment,  a  risk/benefit  assessment  is  not  required. 

Benefits 

The  results  from  this  study  will  increase  the  understanding  of  the  impact  of  automated  systems  on 
the  enhancement  of  CP  140  capability,  especially  the  integration  with  multiple  UAV  control 
systems  on  board. 

A.3.16  Approximate  Time  Involvement 

Each  data  collection  session  is  anticipated  to  take  two  days,  with  the  morning  of  the  first  day  used 
for  training  and  the  rest  of  one  and  half  days  used  for  the  six  experimental  runs  (1A1  ON  and  1A1 
OFF).  Each  day  will  be  at  least  eight  (8)  hours  long  and  it  is  expected  that  the  test  subjects  will 
stay  two  or  three  nights  in  the  Ottawa  area. 

A.3.17  Remuneration 

No  remuneration  will  be  provided  for  involvement  in  this  experiment  as  the  test  subjects 
are  all  serving  members  of  DND.  However,  subjects’  travel  expenses  will  be  reimbursed 
by  DRDC  Toronto. 
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Annex  B  Pre-screening  checklist 


The  following  information  will  only  be  retained  for  the  purposes  of  this  study,  and  will  not  be 
disclosed  to  any  other  individual  or  organization. 


Screening  Questionnaire 

Answer 

1.  Rank? 

2.  Age? 

3.  Gender? 

Male  D 

Female  □ 

4.  Have  you  ever  had  experience  in  UAV  control? 

Yes  □ 

No  □ 

5.  Have  you  ever  had  operational  exposure  to  CP  140? 

Yes  □ 

No  □ 

6.  What  positions  have  you  served  on  CP  140 
if  you  answer  is  Yes  to  question  5? 

Pilot  0 

NAVCOM  □ 

TN  □ 

NASO  a 

ASO  dt0 

Other  U 

7.  Number  of  years  of  your  experience  in  above  positions? 

8.  Estimated  number  of  hours  of  your  experience  in  above  roles? 

9.  Do  you  have  any  difficulties  seeing  colour? 

Y es  O 

No  □ 
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Annex  C  Participants  information  package 


Title  of  Experiment:  Experimental  Assessment  of  Intelligent  Adaptive  Interfaces  for  UAV 

Principle  Investigator:  Dr.  Ming  Hou,  DRDC  Toronto 

Run  Director:  Mr.  Bob  Kobierski,  CMC  Electronics  Inc.  (Ottawa) 

Background:  The  primary  mission  when  developing  a  product  or  system  focuses  on  maximizing 
quality  and  minimizing  cost.  To  accomplish  this  mission,  products  and  systems  are  often 
developed  via  an  iterative  design  cycle,  which  includes:  research  and  development,  definition  of 
requirements  and  specifications,  a  concept/preliminary  design,  the  development  of  a  prototype, 
and  experimental  evaluation. 

After  the  prototype  is  built,  the  design  of  the  product  or  system  is  reviewed  to  gain  feedback  on 
the  design.  Specifically  in  this  study,  an  experimental  evaluation  will  be  conducted  to  exam  the 
efficacy  of  prototype  intelligent  adaptive  interfaces  for  the  control  of  multiple  UAVs  from  an 
airborne  platform.  This  is  the  last  phase  of  DRDC  three-year  project:  Advanced  UAV  Operator 
Interface  Design,  which  will  be  performed  by  CMC  Electronics  Inc  who  is  under  contract  with 
DND.  Based  on  the  results,  recommendations  and  guidelines  will  be  proposed  for  the  design  of 
advanced  UAV  operator  interfaces. 

Task:  The  participants  in  the  study  will  be  required  to  work  together  in  front  of  a  simulated  UAV 
control  unit  which  includes  three  control  stations  for  UAV  pilot,  UAV  operator,  and  TN. 
Through  operator  interfaces  and  associated  input  devices  (e.g.,  trackball,  joystick,  keyboard,  and 
PEP,  etc.),  these  three  individuals  will  work  as  a  UAV  crew  to  search  and  find  a  terrorist  vessel  in 
a  counter-terrorist  scenario  near  the  East  Coast.  The  scenario  runs  for  about  one  hour  with  three 
parts  and  each  part  takes  about  20  minutes.  There  will  be  two  runs  for  the  experiment.  Upon 
completion  of  the  experiment,  the  participants  will  be  required  to  complete  a  debriefing  session. 
Participant  will  be  given  a  short  break  approximately  every  20  minutes.  There  will  be  two 
experimental  sessions,  which  are  approximately  sixty  minutes  in  duration  for  each.  The  entire 
experiment  will  take  about  fifteen  (15)  hours  including  briefing,  training,  conducting  trials,  and 
debriefing.  Participants  are  expected  to  stay  in  Ottawa  area  for  two  nights. 

Benefits  of  this  study:  The  results  from  the  proposed  study  will  increase  the  understanding  of  the 
impact  of  automated  systems  on  the  enhancement  of  CP  140  capability,  especially  the  integration 
with  multiple  UAV  control  systems  on  board. 

Risks  to  Subject:  No  unusual  risks  are  anticipated  other  than  slight  discomfort  in  posture,  vision, 
eyestrain  and  possible  headache. 

Note:  Participants  are  not  expected  to  prepare  for  the  test  by  inquiring  other  crews  who  have 
been  involved  in  the  previous  study  or  by  individual  reading  on  the  subject  of  this  matter. 
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Annex  D  Experimental  instructions 


The  following  briefing  will  be  handed  out  to  the  participants  as  it  appears  in  the  participant 
information  package  (Annex  B)  and  will  also  be  explained  by  the  experimenter  to  the  prior 
beginning  of  the  training  session  and  experimental  session. 

The  participants  (you)  in  the  study  will  be  required  to  work  together  in  front  of  a  simulated  UAV 
control  unit  which  includes  three  control  stations  for  UAV  pilot,  UAV  sensor  operator,  and  TN. 
Through  operator  interfaces  and  associated  input  devices  (e.g.,  trackball,  joystick,  keyboard,  and 
PEP,  etc.),  these  three  individuals  (three  of  you)  will  work  as  a  UAV  crew  to  search  and  find  a 
terrorist  vessel  in  a  counter-terrorist  scenario  near  the  East  Coast.  The  scenario  runs  for  about 
one  hour  with  three  parts  and  each  part  takes  about  20  minutes.  There  will  be  two  runs  for  the 
experiment.  Upon  completion  of  the  experiment,  the  participants  (you)  will  be  required  to 
complete  a  debriefing  session.  Participant  (you)  will  be  given  a  short  break  approximately  every > 
20  minutes.  Since  there  will  be  two  experimental  sessions,  which  are  approximately  one  hours  in 
duration  for  each,  the  entire  experiment  will  take  about  fifteen  (15)  hours  including  briefing, 
training,  conducting  trials,  and  debriefing.  Participants  (you)  are  expected  to  stay  in  Ottawa 
area  for  two  or  three  nights. 
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Annex  E  Consent  form 


Protocol  Number:  L-51 1 

Project  Title:  Experimental  Assessment  of  Intelligent  Adaptive  Interfaces  for  UAV  Operators 
Principal  Investigator:  Dr.  Ming  Hou 

I, _ ,  hereby  volunteer  to  participate  as  a  subject  in  the  study, 

“Experimental  Assessment  of  Intelligent  Adaptive  Interfaces  for  UAV  Operators”.  I  have  read 
the  Protocol  Briefing  Form,  and  have  had  the  opportunity  to  ask  questions  of  the  study 
facilitators.  All  of  my  questions  concerning  this  study  have  been  fully  answered  to  my 
satisfaction.  However,  I  may  obtain  additional  information  about  the  research  project  and  have 
any  questions  about  this  study  answered  by  contacting  Dr.  Ming  Hou  at  (416)  635-2063  or  Mr. 
Bob  Koberiski  at  (613)  592-7400  x  2208 

I  have  been  told  that  I  will  be  asked  to  participate  in  two  experimental  sessions,  which  is 
approximately  sixty  minutes  in  duration  for  each  and  15  hours  for  the  entire  experiment  including 
briefing,  training,  conducting  trials,  and  debriefing. 

I  have  been  told  that  there  are  no  unusual  risks  anticipated.  I  consider  these  risks  acceptable. 
Also,  I  acknowledge  that  my  participation  in  this  study  may  involve  risks  that  are  currently 
unforeseen  by  the  investigators. 

I  understand  that  I  am  considered  to  be  on  duty  for  disciplinary,  administrative  and  Pension  Act 
purposes  during  my  participation  in  this  study.  This  duty  status  has  no  effect  on  my  right  to 
withdraw  from  the  experiment  at  any  time  I  wish  and  I  understand  that  no  action  will  be  taken 
against  me  for  exercising  this  right.  Furthermore,  I  understand  that  if  my  participation  in  this 
study  results  in  a  medical  condition  rendering  me  unfit  for  service,  I  may  be  released  from  the  CF. 

I  have  been  advised  that  there  will  be  no  medical  screening  or  physician  coverage  provided 
during  my  participation  in  the  experiment. 

I  have  been  advised  that  the  experimental  data  concerning  me,  and  the  video  taken  during  my 
trials  will  be  treated  as  confidential  (‘Protected  B’  IAW  CF  Security  Requirements),  and  not 
revealed  to  anyone  other  than  the  investigators  without  my  consent  except  as  data  unidentified  as 
to  source. 

In  the  highly  unlikely  event  of  becoming  ill  during  my  experimental  trial,  I  will  go  with  the 
Investigators  to  seek  immediate  medical  attention  if  either  the  Investigators  or  I  consider  that  it  is 
required.  Every  effort  will  be  made  to  contact  a  family  member  or  the  designated  person 
indicated  below  that  will  be  necessary. 

I  understand  that  I  am  free  to  refuse  to  participate  and  may  withdraw  my  consent  without 
prejudice  or  hard  feelings  at  any  time.  Will  I  withdraw  my  consent,  my  participation  within  the 
experiment  will  stop  immediately. 
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I  understand  that  for  my  participation  in  this  research  project  will  not  be  compensated  for  any 
stress  allowance  for  each  completed  session.  However,  my  TD  expenses  will  be  reimbursed  by 
DRDC  Toronto. 

I  have  informed  the  Principal  Investigator  that  I  am  currently  a  subject  in  the  following  other 

DRDC  Toronto  research  project(s) _ and/or  that  I  am 

participating  as  a  subject  in  the  following  research  project(s)  at  an  institution  other  than  DRDC 
Toronto. 

Volunteer’s  Name: _ 

Signature: _  Date: _ 

Name  of  Witness  to  Signature: _ 

Signature: _  Date: _ 

Family  Member  or  Contact  Person  (name,  address,  daytime  phone  number  &  relationship 


**Note: 

For  military  personnel:  All  military  personnel  must  obtain  their  Commanding  Officer’s  signature 
designating  approval  to  participate  in  this  research  project. 

FOR  SUBJECT  ENQUIRY  IF  REQUIRED: 

Will  I  have  any  questions  or  concerns  regarding  this  project  before,  during,  or  after  participation, 
I  understand  that  I  am  encouraged  to  contact  Defence  R&D  Canada  -  Toronto  (PO  Box  2000, 
1133  Sheppard  Avenue  West,  Toronto,  Ontario,  M3M  3B9).  This  contact  can  be  made  by  surface 
mail  at  these  addresses,  or  in  person,  by  phone,  or  by  email  to  any  of  the  numbers  and  addresses 
listed  below: 

Principal  Investigator 

Dr.  Ming  Hou,  (416)  635-2063,  Ming.Hou@drdc-rddc.gc.ca 
Chair,  DRDC  Human  Research  Ethics  Committee  (HREC) 

Jack  P.  Landolt,  Ph.D.,  (416)  635-2120,  Jack.Landolt@drdc-rddc.gc.ca 


I  understand  that  I  will  be  given  a  copy  of  this  consent  form  so  that  I  may  contact  any  of  the 
above-mentioned  individuals  at  some  time  in  the  future  will  that  be  required. 
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Annex  F  Subjective  measure  rating  forms 


As  noted  above  in  the  protocol,  subjective  ratings  of  performance  are  linked  to  the  hierarchical 
goals  of  the  crew  member  and  will  be  obtained  using  7-point  scales  (see  format  shown  for  SA 
ratings  below). 

Situational  Awareness 

Objective  measures  of  SA  include  a  SAGAT  approach  whereby  the  simulation  is  frozen  and  the 
screens  are  blanked  at  specific  points  in  the  mission.  The  UAV  crew  are  then  required  to  provide 
information  regarding  the  current  status  of  the  various  agents  in  the  mission  (e.g.,  status  of  a  UAV 
asset,  location  and  disposition  of  contacts)  as  well  as  information  regarding  the  predicted  future 
status  of  the  agents  (e.g.,  where  the  contact  will  be  in  1,  5  10  minutes). 


Subjective  ratings  of  Situation  Awareness  will  be  obtained  using  7-point  scales.  An  example  of 
a  7-point  scale  to  index  situation  awareness  of  the  UAV  Pilot  is  as  follows: 


SITUATION  AWARENESS 

Awareness  of: 

very 

low 

moderate 

very  high 

Each  UAV  location 
with  respect  to  the 
CP140; 

1 

2 

3 

4 

5 

6 

7 

NA 

Each  UAV  system 
status; 

1 

2 

3 

4 

5 

6 

7 

NA 

The  heading,  altitude 
and  airspeed  of  each 
UAV; 

1 

2 

3 

4 

5 

6 

7 

NA 

Whether  each  UAV 
is  undertaking  the 
appropriate  (most 
effective)  tasking  for 
the  current  situation; 

1 

2 

3 

4 

5 

6 

7 

NA 

The  current  state  of 
Terrorist  Boat 
Classification  or 
Identification; 

1 

2 

3 

4 

5 

6 

7 

NA 
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Workload  Measure 


As  noted  above  in  the  protocol,  a  PDT  may  be  used  to  obtain  objective  measures  of  workload. 
Subjective  ratings  in  the  form  of  a  modified  NASA  TLX  will  be  used.  These  rating  forms  will  be 
tuned  for  each  crew  member  and  for  each  of  the  crew  members’  critical  tasks.  Since  the 
relatively  small  sample  size  in  this  experiment,  analyses  of  cross/inter-correlations  are  not 
justified  and  paired  comparisons  will  not  be  performed. 


RATING  SCALE  DEFINITIONS 

Title 

Endpoints 

Descriptions 

MENTAL  DEMAND 

Low/High 

How  much  mental  and  perceptual  activity  was  required 
(e.g.,  thinking,  deciding,  calculating,  remembering, 
looking,  searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving? 

PHYSICAL 

DEMAND 

Low/High 

How  much  physical  activity  was  required  (e.g.,  pushing, 
pulling,  turning,  controlling,  activating,  etc.)?  Was  the 
task  easy  or  demanding,  slow  or  brisk,  slack  or 
strenuous,  restful  or  laborious? 

TEMPORAL 

DEMAND 

Low/High 

How  much  time  pressure  did  you  feel  due  to  the  rate  or 
pace  at  which  the  tasks  or  task  elements  occurred?  Was 
the  pace  slow  and  leisurely  or  rapid  and  frantic? 

EFFORT 

Low/High 

How  hard  did  you  have  to  work  (mentally  and 
physically)  to  accomplish  your  level  of  performance? 

PERFORMANCE 

Good/Poor 

How  successful  do  you  think  you  were  in  accomplishing 
the  goals  of  the  task  set  by  the  experimenter  (or 
yourself)?  How  satisfied  were  you  with  your 
performance  in  accomplishing  these  goals? 

FRUSTRATION 

LEVEL 

Low/High 

How  insecure,  discouraged,  irritated,  stressed  and 
annoyed  versus  secure,  gratified,  content,  relaxed  and 
complacent  did  you  feel  during  the  task? 
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Instructions:  Using  the  rating  scales  below  and  the  definitions  provided,  mark  each  scale  to 
indicate  the  degree  of  workload  you  experienced  while  completing  the  indicated  task. 

Operator:  UAV  Pilot 

Task:  Establish  the  Mini  UAV  in  a  position  to  allow  the  UAV  Sensor  Operator  to  use  the  EO 
suite  to  classify  an  unknown  fishing  boat. 

MENTAL  DEMAND 

Low  High 

PHYSICAL  DEMAND 

I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Low  High 

TEMPORAL  DEMAND 

I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Low  High 

EFFORT 

Low  High 


PERFORMANCE 

Good  Poor 

FRUSTRATION 

Low  High 
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List  of  abbreviations 


ANOVA 

Analysis  of  Variance 

CF 

Canadian  Forces 

CFB 

Canadian  Forces  Bases 

CHOGM 

Commonwealth  Heads  of  Government  Meeting 

CTS 

Completion  Time  for  Critical  Task  Sequence 

DMS 

Data  Management  System 

DND 

Department  of  National  Defence 

DRDC 

Defence  Research  &  Development  Canada 

EO 

Electro- Optic 

GUI 

Graphical  User  Interface 

HGA 

Hierarchical  Goal  Analysis 

HUD 

Heads-Up-Display 

IAI 

Intelligent  Adaptive  Interface 

MALE  UAV 

Medium  Altitude  Long  Endurance  Uninhabited  Aerial  Vehicle 

MH 

Maritime  Helicopter 

MOC 

Maritime  Operations  Centre 

NATO 

North  Atlantic  Treaty  Organization 

NAVCOM 

Navigator  Communicator 

nm 

nautical  miles 

OPI 

Office  of  Primary  Interest 

PACT 

Pilot  Authorization  and  Control  of  Tasks 

PEP 

Programmable  Entry  Panel 

PVI 

Pilot- Vehicle  Interface 

R&D 

Research  &  Development 

RTB 

Research  Test  Bed 

SA 

Situation  Awareness 

SAGAT 

Situation  Awareness  Global  Assessment  Technique 

SE 

Synthetic  Environment 

SME 

Subject  Matter  Expert 

STANAG 

Standardization  Agreement 
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TACPLOT 

Tactical  Plot 

TLX 

Task  Load  Index 

TN 

Tactical  Navigator 

UAV 

Uninhabited  Aerial  Vehicle 

uo 

UAV  Sensor  Operator 

UP 

UAV  Pilot 

VTUAV 

Vertical  Take-Off  UAV 
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project  conducted  by  DRDC  Toronto.  This  project  investigated  the  efficacy  of  IAIs  in  a  multi- 
Uninhabited  Aerial  Vehicle  (UAV)  scenario.  The  IAI  was  modelled  as  part  of  the  UAV  tactical 
workstations  found  in  a  maritime  patrol  aircraft.  In  the  first  phase  of  the  project,  a  performance 
model  was  developed  to  compare  the  difference  in  mission  activities  with  and  without  IAI  agent 
aids.  The  simulation  results  revealed  that  the  control  of  multiple  UAVs  is  a  cognitively  complex 
task  with  high  workload.  With  the  augmentation  of  automation  agents,  operators  could  continue 
working  under  high  time  pressure,  resulting  in  critical  tasks  being  achieved  in  reduced  time.  To 
further  test  the  effectiveness  of  IAIs  and  validate  the  simulation  results,  a  prototype  IAI  multi¬ 
agent  experimental  environment  was  implemented  for  an  empirical  study.  Six  IAI  agent 
function  groups  have  been  integrated  into  the  UAV  operator  interfaces.  Operator’s  performance 
was  examined  with  and  without  IAIs  under  three  different  workload  conditions.  The  results 
from  both  objective  and  subjective  measures  verified  the  findings  of  the  simulation  research. 
IAIs  facilitated  a  significant  reduction  in  workload  and  an  improvement  in  situation  awareness. 
This  research  also  developed  preliminary  guidance  on  designing  IAI  systems. 

Un  probleme  qui  se  pose  en  ce  qui  conceme  la  commande  de  plusieurs  engins  telepilotes  est  la 
gestion  de  la  masse  d’ informations  necessaires  pour  appuyer  la  prise  de  decision  efficace.  De 
Tavis  des  operateurs  d’engins  telepilotes,  l’amelioration  des  interfaces  operateur  entrainerait  des 
gains  importants  au  niveau  des  performances  et  de  l’efficacite  des  systemes.  Divers  niveaux 
d’automatisation  ont  ete  suggeres  pour  resoudre  le  probleme,  dont  Tutilisation  d’interfaces 
adaptatives  et  intelligentes  (IAI)  pour  l’aide  a  la  decision.  En  dotant  les  postes  de  commande  des 
engins  telepilotes  de  groupes  de  fonctions  d’automatisation,  les  IAI  ont  pour  but  de  gerer 
Tinformation  dynamiquement  et  de  foumir  la  bonne  information  aux  bonnes  personnes  au  bon 
moment,  pour  appuyer  la  prise  de  decision  efficace.  Les  travaux  decrits  dans  le  present 
document  sont  l’aboutissement  d’un  projet  de  trois  ans,  realise  par  R  &  D  pour  la  defense 
Canada,  portant  sur  Pefficacite  des  IAI  dans  un  scenario  de  commande  de  plusieurs  engins 
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d’execution  des  taches.  Un  prototype  d’environnement  experimental  d’lAI  a  ete  mis  en  oeuvre 
pour  une  etude  empirique  a  intervention  humaine.  Les  resultats  de  la  simulation  et  de 
l’experience  ont  montre  que  la  commande  de  plusieurs  engins  telepilotes  est  une  tache 
complexe  sur  le  plan  cognitif  avec  charge  de  travail  elevee.  Avec  l’ajout  d’agents 
d’automatisation,  les  IAI  ont  favorise  une  baisse  appreciable  de  la  charge  de  travail  et  une 
amelioration  de  la  connaissance  de  la  situation.  Les  operateurs  peuvent  continuer  a  travailler 
sous  de  fortes  contraintes  de  temps,  et  des  taches  critiques  peuvent  etre  executees  en  moins  de 
temps  qu’avec  des  interfaces  classiques. 
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